Developers are builders by nature (and profession), so many take pride in building their own solutions to problems from first principles, often using tools developed in open-source projects. So as Internal Developer Portals (IDPs) increased in popularity, it should come as no surprise that interest in open source IDPs increased in kind. While we may not yet have a fully opensource IDP platform, in this blog we’ll cover the open source components and platforms used to build IDPs from scratch.
What is an internal developer portal?
An IDP serves as a central hub for engineering teams to track, record and improve tools and processes used to build high-quality software. From services and APIs to Kubernetes clusters and data pipelines, IDPs abstract away complexities to enhance security and production readiness drive production speed and quality using data from your existing tools. They are a progression from Internal Developer Platforms —adding critical data aggression and standardization platforms needed to drive reliability in software deployments.
Portals came about in response to proliferation of developer tools and complexities in the development environment, from technology changes like containerization and microservices, to an explosion of SDLC tooling that includes everything from code editors to IDE software. Each new tool or technique theoretically makes developers' jobs easier, but with often inconsistent ways of modeling data across them, and dispersed across a wide organization with many distributed teams, these changes have created an environment in desperate need of standardization. Developer Portals help standardize data, requirements, and best practice.
Benefits of an internal developer portal
Portals act as an interface with SDLC tools and resources, as well as creating and standardizing golden paths to enforce best practice engineering. They map all entities, from APIs to microservices, to improve software development by enforcing standards and expectations at scale. The long-term benefits go beyond centralising docs and making processes extensible: portals empower developers with self-service solutions that reduce dependencies on DevOps and associated bottlenecks.
By acting as a central repository of best practice across the engineering team, portals collapse any inconsistencies in culture or approach that can emerge from work taking place in silos. This standardizes development in line with defined best practice within your organization, which increases productivity. Stripping away operational tasks, complexity and the need for excessive communication increases the velocity of development.
Definitions are important here: velocity represents speed in a given direction, so teams can have considerable speed without velocity when they are not aligned or they duplicate work. Portals solve this problem. They also represent an accessible repository of your engineering culture, practices and philosophy, which radically reduces onboarding for new recruits and allows them to start adding almost value immediately. These repositories are known as software catalogs.
What is an open source developer portal?
An open source developer portal is an IDP that is built on open source technology. These cost nothing to the user in terms of seats and lisences, and developers can often contribute to the underlying technology, giving back to the community while enhancing their experience. They do require greater investment in setup and maintenance, so costs should be measured according to demands on internal capacity of platform engineering and front-end resources, and opportunity costs of diverting attention of these teams from other projects.
The first widely adopted open source internal developer portal sprang from an internal project known as Backstage within Spotify’s engineering team. Backstage’s original form was a service catalog that enabled their teams to more easily understand and track service ownership, documentation, and dependencies across their organization.
Elements of a service catalog
Pre-dating but formative to our modern definition of IDPs, service catalogs were the original way (or... the one-step-up from spreadsheet way) organizations consolidated service information in microservice architectures. With this solution, members of distributed teams could quickly find and use services they might not have otherwise known existed. Service catalogs have since expanded to the more general term of software catalogs—to account for things like APIs, pipelines, clusters, and more.
To qualify as a service (or software) catalog, a solution must contain the following:
- Ownership: Catalogs serve as a central registry for software allowing organizations to view, track, and ensure upkeep over time. But without ownership information, services may fall out of alignment with best practice. Surfacing ownership is information is the first step in ensuring software accountability.
- Basic scaffolding: Most catalogs enable the creation and centralization of shared templates the rest of the engineering team can use to build and deploy new software according to best practice, while using boilerplate code that reduces effort and risk.
- Dependency management: While many organizations now have standalone tooling for dependency management, software and service catalogs should persist those relationships from those tools for easy investigation from a central location that lets users view ownership, language, and version information in one place.
What is Spotify Backstage?
Spotify developed a software catalog internally called System Z back in 2014, which grew to act as a jumping off point for frontend UIs of engineering tools. In 2017, System Z was comprehensively rewritten to give us Backstage as we know it today, and in 2020 it was open sourced and accepted to the Cloud Native Computing Foundation (CNCF).
Backstage is a DIY portal that can be either self-hosted or run by a vendor. Its core feature remains the software catalog, which tracks ownership of every entity within the organization (service, API, domain). It also includes a scaffolder that creates software templates for new services and components, collecting best practice from developers before incorporating these into existing codebase templates. As well as this Backstage hosts technical documentation, and enables plugins to enhance security, audits, operational tools and more.
Backstage has a range of use cases and is typically used to improve velocity, efficiency and security of code development. By acting as a system of record and abstracting away some complexity, it allows developers to leverage existing tools and best practice to improve software development across the board.
Putting open source software at the core of your operations also comes with its challenges, mostly related to the DIY nature of open source and the associated work that comes with it. It helps to consider the costs in terms of lost capacity and opportunity cost of the internal team's time, versus the cost of paying for a managed IDP.
Real costs and opportunity costs
The first consideration when pricing an open source developer portal is installation costs. Backstage is code-based, so standing it up means customizing open source and YAML files. It also doesn't ship prebuilt containers, meaning installation requires Node.js and Yarn. Building and maintaining this open source IDP requires a dedicated platform engineering team, and according to one founder self-hosted Backstage teams spend up to 30% of their time upgrading their instances.
A lot of the value portals provide comes from their ability to integrate live data from across an organization into scorecards and reporting. Because this data is integrated via third party plugins, each needs to be individually configured, and functionality for features like search can vary. This integration is an ongoing burden, as third party applications will require regular updates and fixes on top of changes associated with evolving demands from developers.
Maintenance costs are also a factor for DIY portals like Backstage, as the open source components involved will also need frequent updates. Security is more hands on when using open source, which takes up additional resources. Onboarding and training are also a significant cost, as there is no third party to support with queries or troubleshooting.
With Backstage, much of the value that is added comes during the installation and customization phase when platform engineers do the detailed work needed to adapt the software to your organization's specific needs. Following this, there is onboarding, training, and an internal campaign to promote uptake and demonstrate the value of the new portal. All of this increases the time-to-value of the installation, causing an initial dip in productivity with the expectation of future improvement.
To summarise, open source portals involve no costs for seats or licences, but require considerable time and capacity from the internal team to get right. You will need a dedicated platform engineering team to operate the software, an upfront cost to standing the portal up and training up users, and ongoing maintenance to keep it running. All of this is a drain on resources, and even the best internal teams are highly unlikely to replicate the quality of analytics that are available from third party portals.
Open source developer portals vs. “managed” developer portals
The alternative to building an open source developer portal is to pay for a managed developer portal off-the-shelf. These are proprietary products that are straightforward to adopt: just as portals abstract away the backend work around tools and services for developers, managed portals abstract away the building and maintenance requirements for platform engineers.
Managed portals come with extensive support and documentation, which helps with implementation. Buying your portal out of the box reduces time-to-code and eliminates installation work, allowing you to get straight to optimizing it for your team rather than building it like a product.
Most managed portal providers will do the hard work to make sure that plugins are convenient and effective, with all necessary functionality built in from the start. This can even extend to internal data sources, making the managed portal a comprehensive system of record with access to all relevant data.
Finally, managed portals come with robust automation and analytics capabilities from day one, something that would require significant time and expertise to build from open source tools. These provide the highest quality insights without the need to build or integrate tools, or to hire engineers to take care of this.
How can Cortex help?
If you think building and maintaining an open source internal developer portal is a false economy, or you are looking to transition from using Backstage to a managed SaaS portal, Cortex can help. The Cortex portal also offers a microservices catalog, but extends this to include all software entities and dependencies with constant up to date ownership.
Cortex also offers a scaffolder to help you bootstrap new entities quickly and in line with established best practice, which allows developers to self-serve and reduces time to code. This draws from existing templates and because it is automatically integrated into the catalog, it creates a central repository that ensures consistency and tracks usage to continue to refine best in class development.
Cortex's scorecards also represent another major advantage over open source alternatives, by offering a quantitative assessment of your software entities’ health, performance, and compliance, aligning development and business teams on common measurable objectives. Unlike Backstage, which requires you to define and implement your own health checks and performance metrics, Cortex's Scorecards come predefined with industry best practice measures.
These scorecards automatically aggregate and analyze data from across your software ecosystem and display it on real-time dashboards to provide a true reflection of the health of your entire software estate. This empowers teams to proactively mitigate risks and enforce best practices consistently, leading to superior software quality and faster time to market.
In addition to the aforementioned features, Cortex differentiates itself with its array of robust plugins. While open source plugins need to be individually developed, integrated and maintained, Cortex offers a suite of ready-to-use plugins that seamlessly integrate with your existing workflows. These bridge the gap between multiple software entities, as well as internal and external data sets, drastically reducing the integration overhead and offering a unified view of your entire software ecosystem.
This not only streamlines the developer experience but also enhances visibility for all stakeholders, improving decision-making and enabling a more proactive approach to software health and performance.
To learn more about internal developer portals check out the Cortex white paper, take a self-guided tour, or book a conversation with us here.
