source

April 25, 2022 08:09 ET | Source: Fortune Business Insights Fortune Business Insights
Pune INDIA
Pune, India, April 25, 2022 (GLOBE NEWSWIRE) — The global document management system market size was USD 5.00 billion in 2021 and reached USD 5.55 billion in 2022. The market is anticipated to reach USD 16.42 billion by 2029, exhibiting a CAGR of 16.8% during the forecast period. The rising demand for paperless government and offices due to the extensive adoption of cloud services is expected to propel the market development. Fortune Business Insights™ provides this information in its report titled “Document Management System Market Growth, 2022-2029.”
A document management system is a solution developed to systematically manage documents and files and simplify data management. The rising demand for paperless government and offices may enhance the market growth. Further, the extensive adoption of cloud-based services may enhance the product adoption. These factors may propel the industry’s growth in the coming years.
Key Industry Development


Request a Sample Copy of the Research Report: https://www.fortunebusinessinsights.com/enquiry/request-sample-pdf/document-management-system-market-106615


Report Scope:


Drivers and Restraints
Robust Demand for Workplace Efficiency to Enhance Market Growth
The incorporation of advanced technology such as artificial intelligence, real-time tracking solutions, and cloud computing solutions is expected to surge the product demand. For example, eGrove Systems Corporation announced an integrated advanced agile document and time tracking project management. This factor increased workplace efficiency by using advanced software solutions. Further, incorporating the software enables companies to manage the workplace environment and achieve their goals. These factors may propel the document management system market growth.
However, increasing data privacy concerns and regulatory compliances may hinder market growth.


Click here to get the short-term and long-term impact of COVID-19 on this Document Management System Market.
Please visit: https://www.fortunebusinessinsights.com/document-management-system-market-106615


Regional Insights

Presence of Major Players to Propel Market Progress in North America
North America is expected to dominate the document management system market share due to the presence of several major players. The market in North America stood at USD 2.25 billion in 2021 and is expected to gain a huge portion of the global market share. Further, the presence of a developed digital infrastructure is expected to boost the industry progress.
In Asia Pacific, the rising adoption of DMS solutions by government, manufacturing, and other sectors is expected to boost the document management system adoption. These factors may propel the market growth.
In Europe, rising investments in digital platforms may boost the adoption of the document management system. Further, rising digital platform investments are expected to boost industry progress.






Segments
By component, the market is segmented into solution and services. As per deployment, it is bifurcated into cloud and on-premises. Based on organization size, it is clubbed into large enterprises, and small and medium enterprises. By industry, it is classified into BFSI, IT and telecommunication, government, manufacturing, retail, healthcare, and others. Regionally, it is classified into North America, Europe, Asia Pacific, Middle East & Africa, and South America.
Competitive Landscape
Players Announce Novel Services to Boost Brand Image
The prominent players operating in the market announce novel services to enhance their sales and boost brand image. For example, Google LLC announced an AI-based Lending DocAI service for the mortgage industry. The AI tool helps several mortgage companies in speeding up their document processing. It helps automate routine document reviews by extracting the data required. It is a civilized document that may enable the company to boost its brand image. Further, companies adopt research and development, mergers, acquisitions, and expansions to boost their annual revenues and global market position.


Quick Buy – Document Management System Market:
https://www.fortunebusinessinsights.com/checkout-page/106615


Report Coverage
The report provides a detailed analysis of the top segments and the latest trends in the market. It comprehensively discusses the driving and restraining factors and the impact of COVID-19 on the market. Additionally, it examines the regional developments and the strategies undertaken by the market’s key players.
COVID-19 Impact
Rising Dependence Upon Digitization to Foster Market Growth
This Document Management System Market is expected to be negatively affected during the COVID-19 pandemic because of the rising dependence on digitization. The alarming spike in COVID-19 cases leads to restrictions on manufacturing and the closure of activities. Companies focus on developing digital infrastructure to continue their activities and enhance their annual revenues. The accumulation of digital data loads leads to the adoption of effective data management, thereby enhancing the adoption of the product. These factors may propel the market progress during the pandemic.
Companies Profiled in the Document Management System Market Report


Have Any Query? Ask Our Experts: https://www.fortunebusinessinsights.com/enquiry/speak-to-analyst/document-management-system-market-106615


Major Points of Table:
TOC Continued…!
About Us:
Fortune Business Insights™ offers expert corporate analysis and accurate data, helping organizations of all sizes make timely decisions. We tailor innovative solutions for our clients, assisting them to address challenges distinct to their businesses. Our goal is to empower our clients with holistic market intelligence, giving a granular overview of the market they are operating in.
Contact Us:
Fortune Business Insights™ Pvt. Ltd.
US :+1 424 253 0390
UK : +44 2071 939123
APAC : +91 744 740 1245
Email: sales@fortunebusinessinsights.com

source

source

Sign up
Sign In
Sign up
Sign In
DataDrivenInvestor
Jun 17
Save
While you are creating, editing, or sending digital documents, it is necessary to capture an online signature so that the document can be authenticated. For effectively capturing a legally binding electronic signature online, there is a need for an electronic signature service.
Without the implementation of an online signature software, you will end up exposing your company to legal troubles. One of the top electronic signature software that are used by organizations commonly is called DocuSign.
Click to the Image to Check Price
It is available at a nominal fee per month and ends up being a low-risk method to send a couple of documents per month while also getting an understanding of how online document signing actually operates.
As said, DocuSign is not the only choice that there is for an online signature software.
However, I’m just diverting your attention with this statement, DocuSign competition is a burning issue in the electronic signature software industry. Each DocuSign competitor has been trying to beat it by facilitating the user’s with same features in less budget.
Furthermore, all the DocuSign competitors that have been mentioned below provide free trial with some features. You can utilize for a days, if you are not satisfied with them. You can easily switch it and try another alternative to DocuSign.
There are multiple competitors and alternatives to DocuSign that are available in the market and they make a great choice. In this blog, I have listed top DocuSign competitors and alternatives that can make your selection convenient.
Let we start understanding of free DocuSign alternative one by one. Each DocuSign Competitor’s pricing have denoted based on the basic features that they offer. You may find out more about business and enterprise planning by visiting the software’s website.
➽ SignNow — 1st Free DocuSign Competitor & Alternative
➽ CEO — Borya Shakhnovich
➽ Mobile App: iOS | Android
➽ Location — Brookline, Massachusetts
In the list of DocuSign’s competitors, SignNow is one of the top Docusign competitor and electronic signature software for small businesses that comes with features needed to sign and send documents. It helps in generating agreements, automating and streamlining processes, accessing payments, and managing documents.
This application has reusable templates that help in simplifying the process of sending documents and saving time. When it comes to workflows, SignNow allows you to organize documents into groups and send them based on the roles of receivers.
With SignNow, it is also possible to set different actions after signing has been completed.
➽ Wesignature — Free DocuSign Competitor & Alternative
➽ CEO — Ryan Pegram
➽ Mobile App: None, web-based only
➽ Location — Thornton, Colorado
WeSignature is one of the best document signing software and most widely chosen electronic signature software at the present time.
Many professionals have been using this application for their personal and professional use which makes it the best DocuSign competitor.
It is a simple, efficient, and effortless software for signing documents as it enables individuals and organizations to sign a wide range of online documents.
Once you adopt the WeSignature application, you can sign the documents, fill up paperwork, and follow up with the receivers regularly.
It is an application that has consistently proven itself to be the best electronic signature service for small businesses.
Once you start using WeSignature, you will be surprised at the reduction of turnaround time from a couple of days to just a few minutes.
In addition, it also enables the organizations to send multiple documents to people all at once.
➽ Signaturely — Free DocuSign Competitor & Alternative
➽ CEO — Will Cannon
➽ Mobile App: None, web-based only
➽ Location — 340 S Lemon Ave Ste 1760 Walnut, CA 91789
In another competitors to DocuSign, Signaturely is well known esignature software. It is also preferred by many people who are looking for simple methods to get their documents signed in a legal manner. Signaturely has proven itself to be a great alternative to DocuSign because of its simplicity.
Signaturely is easy to use and also makes online document signing simple. The reason why Signaturely stands out is that it focuses on eliminating the features.
It lays special emphasis on cutting down all unnecessary steps so that it becomes easy to get your documents signed.
➽ CocoSign — Free DocuSign Competitor & Alternative
➽ CEO — Stephen Curry
➽ Mobile App: None, web-based only
➽ Location — Singapore
Over the past few years, CocoSign has emerged as an extremely renowned online signature platform. It is one of best DocuSign competitor and is used for sending, signing, saving, and accessing documents online.
It is capable of automating business processes by closing deals quickly, safely, and legally.
CocoSign enables users to choose a free trial for understanding how the platform should work and how useful it can be. It is easily the best place for online signatures as it improves businesses by automating significant parts of business deals. It is empowered with multiple applications, integrations, APIs, and industry-specific solutions.
CocoSign allows you to get signatures digitally without facing problems in managing paperwork. It provides a user-friendly, digital, and integrated experience for creating e-signatures.
In addition, it also offers cross-platform functionality and can be accessed anywhere. People use it because it is safe, legally compliant, and efficient
➽ HelloSign — Free DocuSign Competitor & Alternative
➽ CEO — Joseph Walla
➽ Mobile App: None, web-based only
➽ Location — San Francisco, CA
HelloSign is another online electronic signature software that is also understood to bring a wide range of features to the market. It is great with customer service, customization, and flexible pricing as well.
It also comes with a great API that enables you to embed and brand the signing options in the online documents.
This is an electronic signature company that is also compliant with all of the major online signature laws while offering an array of extensions and integrations.
It is an application owned by Dropbox and comes with powerful integration along with many tools such as Google Suite, Gmail, and more.
➽ Adobe Sign — Free DocuSign Competitor & Alternative
➽ CEO — Shantanu Narayen
➽ Mobile App: iOS | Android
➽ Location — San Jose, CA
Adobe Sign is a feature-rich among DocuSign competitors. It is an online signature platform that provides you with the power to manage the workflows from any location and device. Many people use this app because of the seamlessness that it is capable of offering in electronic document signing.
Adobe Sign is an application that is known for its wide integration with third-party tools along with an added focus on global compliance.
It is full of features for both electronic and digital signatures. Many professionals have been choosing Adobe Sign for their personal and professional use.
➽ PandaDoc — Free DocuSign Competitor & Alternative
➽ CEO — Mikita Mikado
➽ Mobile App: iOS | Android
➽ Location — San Francisco, California
Yet another top competitors to DocuSign is PandaDoc which is very well known for offering a streamlined user interface and ease of use.
This is an online signature tool that is known for providing a streamlined user interface and ease of use.
It is an e-signature tool that comes with great assistance in document management.
PandaDoc comes with a drag and drops integration, automated workflow, and audit history as well. It has multiple integrations including CRM, file storage applications, and payments.
If you are looking for an effective solution for the management of contracts then PandaDoc is worth giving a shot.
➽ RightSignature — Free DocuSign Competitor & Alternative
➽ CEO — Daryl Bernstein, Cary Dunn, and Jonathan Siegel
➽ Mobile App: None, web-based only
➽ Location — Fort Lauderdale, FL
Next DocuSign competitor, RightSignature is a perfect alternative to DocuSign that comes with a wide range of integrations as an important part of the e-signature process. It specializes in making document signing a simple process.
The users can upload online documents to RightSignature with a drag and drop tool for placing signature fields inside the document.
Once this happens, the users can send the document through an email to the customer for an optimized online signing experience. Using RightSignature offers plans for individuals and enterprise-level users as well. It provides features that are completely different as is the cost.
RightSignature also treats uploaded documents such as a locked PDF while enabling the users to drag the signature fields on the top of the page.
The custom branding with RightSignature is more like a white labeling feature as opposed to a branding kit.
➽ SignWell — Free DocuSign Competitor & Alternative
➽ CEO — Martin Holmstrom
➽ Mobile App: None, web-based only
➽ Location — Portland, Oregon
SignWell is another one of the competitors to DocuSign that is a cost-effective and user-friendly electronic signature application used by many businesses. It helps in eliminating many hours from the usual document signing process and is also compliant with e-signature laws.
This application comes with a free plan that also includes various features such as document tracking, flexible workflows, and reminders. This is an application that is being used by a wide range of people in recent times.
➽ SignEasy — Free DocuSign Competitor & Alternative
➽ CEO — Sunil Patro
➽ Mobile App: iOS | Android
➽ Location — Brookline, Massachusetts
SignEasy is yet another top recommendation for many people. It is one of the best electronic signing software for personal use. You can sign up with a free trial and you can instantly begin by uploading documents, preparing them for signatures, and sending them.
SignEasy comes with wide integration support and also works within your favorite applications. You can open a document with Gmail, sign it, and then send it without any stress.
Finally, you can also take benefit of many features such as automated reminders, tracking, and signing sequences.
➽ Eversign — Free DocuSign Competitor & Alternative
➽ CEO — Julian Zehetmayr
➽ Mobile App: None, web-based only
➽ Location — Wien, Wien
The another choice in the list for DocuSign competitors is Eversign. It is a great solution for all users who need legally binding online signatures but are not looking to break the bank with a high fee.
Eversign is a cost-effective choice that comes with the ability to send many documents per month without an added fee.
The basic features that Eversign offers are all included in audit trails, contract management, and app integrations.
The businesses that are looking to onboard more users or seeking additional perks such as in-person signing can exist without any extra price tag.
➽ DigiSigner — Free DocuSign Competitor & Alternative
➽ CEO — Jessica Kelly
➽ Mobile App: None, web-based only
➽ Location — 700 N Valley St Suite B Anaheim, CA 92801
DigiSigner is a cloud-based electronic signature software and one of the best DocuSign competitor that focuses on speed, affordability, and convenience of use.
Using the service, businesses and people can sign contracts and agreements from any location in the world, regardless of their location.
DigiSigner is compatible with a wide range of devices, including laptops, tablets, smartphones, and more.
All main e-signature laws, such as ESIGN, UETA, and European eIDAS, are met by DigiSigner.
DigiSigner’s signatures are legally binding and can be used in a court of law.
➽ SIGNiX — Free DocuSign Competitor & Alternative
➽ CEO — Jay Jumper
➽ Mobile App: None, web-based only
➽ Location — Chattanooga, TN
A next DocuSign competitor is SIGNiX, which makes it easy for partners in highly regulated industries like real estate, wealth management, and healthcare to use digital signature and online notarization software together.
There are no costs or risks to using the patented SIGNiX FLEX API.
It allows partners to offer military-grade cryptography, enhanced privacy, and permanent legal evidence of a true digital signature without having to deal with paper-based processes.
➽ Scrive — Free DocuSign Competitor & Alternative
➽ CEO — Viktor Wrede
➽ Mobile App: None, web-based only
➽ Location — Stockholm, Stockholm County
As soon as Scrive was started in 2010, it quickly became a leader in the Nordic e-sign market and take place in the list of top DocuSign competitors.
Today, more than 6000 customers in 40+ countries use Scrive to speed up their onboarding and agreements processes with solutions that use electronic signatures and IDs.
As a trusted digitalization partner, Scrive helps businesses of all sizes, even those in highly regulated industries, move forward with their digital transformations.
This includes improving customer experience, security, compliance, and data quality. Scrive is based in Stockholm and is owned by Vitruvian Partners. It has more than 200 employees.
➽ Secured Signing Software — Free DocuSign Competitor & Alternative
➽ CEO — Gal Thompson
➽ Mobile App: None, web-based only
➽ Location — 800 W. El Camino Real, Suite 180 Mountain View CA 94040
Secured Signing Software is a cloud-based service for managing electronic signatures. It works with businesses of all sizes in a wide range of industries, including finance, education, and real estate.
Users can sign documents digitally, send email invitations to complete documents, and make forms with the help of these tools.
Secured Signing lets people send invitations to a group of people. Businesses can set up reminders and add extra fields to documents to help them.
The solution lets users add electronic signatures to word documents and set up approval processes. It may also let recipients add or change text before they sign.
Secured Signing works with Salesforce, Realme, and Microsoft Dynamics 365. Face-to-face signing lets customers use an SMS code to prove their identity and then sign on the screen.
In the dashboard, users can see how long it will take to sign a form, as well as download and store signed copies of the form.
If you want to use Secured Signing’s services, you can pay for them each month or pay as you go.
Customer service is available through an online ticketing system, an online knowledge base, phone, and email, among other ways.
➽ eSign Genie — Free DocuSign Competitor & Alternative
➽ CEO — Mahender Bist
➽ Mobile App: None, web-based only
➽ Location — Cupertino, California
This electronic document signing service is actually quite expensive, despite the fact that it appears to be a bargain at first glance. Esign Genie is a splendid choice among top notch DocuSign competitors, despite its low price, is packed with capabilities that make the e-signing process more convenient for both signers and organizations.
The process of connecting to eSign Genie is as simple as creating a network connection and using the form-signing tools.
eSign Genie can help you collect document signing for a fraction of the cost of competing products like PandaDoc and GetAccept.
When it comes to the business, pay-as-you-go is one of the most intriguing aspects. This package also includes the ability to sign papers in person and assign signers, two features that are often reserved for more expensive e-signature services.
➽ SignRequest — Free DocuSign Competitor & Alternative
➽ CEO — Geert-Jan Persoon
➽ Mobile App: None, web-based only
➽ Location — Amsterdam, Noord-Holland
SignRequest appears to offer a lot of capability and customizability for senders who need to transmit multiple papers each month. When creating papers for multiple signers, the professional plan comes with features including a post-signature landing page and the ability to alter the document signing sequence.
This platform is ideal for small-business owners who don’t need to transmit a lot of documentation each month, and it offers everything you need.
You can easily see what paperwork has yet to be completed and what documentation has already been finished thanks to SignRequest’s document management features.
When it comes to creating templates, collecting signer attachments, and selecting the authentication mechanism your signatures may use, SignRequest is the best document signing software for businesses or anyone searching for a quick and easy method to sign.
➽ DrySign — Free DocuSign Competitor & Alternative
➽ CEO — Ron Cogburn
➽ Mobile App: None, web-based only
➽ Location — 2701 E. Grauwyler Road Irving, TX 75061, USA
Due to its cloud-based best electronic signing software for company, it may assist speed up internal and external sign-offs, reduce the need for paper procedures, and increase team efficiency.
It is possible to use DrySign with any of the following cloud storage services: Google Drive, Dropbox, OneDrive, and CRM. It can be utilized on a variety of platforms, including PCs, laptops, and even handheld devices.
The ESIGN Act and the UETA are among the electronic signing regulations that this method complies with. DrySign provides an audit trail, multi-factor authentication, and smart tracking to assist businesses reduce the risk connected with their transactions.
All documents and activity can be viewed from the dashboard. Multiple signatures can be requested, automated notifications can be set up, changes can be viewed in real time, document fields can be altered, and bulk files can be uploaded.
➽ KeepSolid Sign — Free DocuSign Competitor & Alternative
➽ CEO — Vasiliy Ivanov
➽ Mobile App: None, web-based only
➽ Location — Bronx, New York
Various documents, such as contracts, transactions, and other agreements, can be signed electronically with KeepSolid Sign. In either case, it is possible to set up this product.
As compared to DocuSign, KeepSolid Sign allows you to sync documents between several platforms, including PCs, tablets, and cellphones. The data in the system is encrypted with AES-256. Apps for iOS and Android devices are also available from the company.
Signing and annotating documents without being online is feasible, and the changes you make are saved. In order to keep track of the progress of a document, the service provides an activity dashboard that may be used by the user.
➽ Formstack Sign — Free DocuSign Competitor & Alternative
➽ CEO — Chris Byers
➽ Mobile App: iOS | Android
➽ Location — Fishers, IN
In the DocuSign competition, Formstack Sign is an e-sign document platform built inside the Formstack data management system, allows users to sign documents electronically.
Filling out surveys and applying for jobs are just two examples of online forms that Formstack Sign is well-suited to handle because of its accessibility and signature automation features.
It is possible to trace the origins of the company back to Ade Olonoh, who founded FormSpring on February 28th, 2006. FormSpring predated FormStack.
Initially, it was designed to be an online form builder that also provided workflow management solutions for businesses like higher education and marketing.
An added benefit is that the integration of more than 50 web applications such as customer relationship management (CRM), email marketing (email marketing), payment processing and document management is available to Formstack customers who don’t have programming or software abilities.
➽ ZorroSign — Free DocuSign Competitor & Alternative
➽ CEO — Shamsh Hadi
➽ Mobile App: iOS | Android
➽ Location — Phoenix, AZ
Document-based transactions, such as payroll and employee onboarding, can be managed with ZorroSign. It is a cloud-based electronic signature and digital transaction management solution.
ZorroSign uses proprietary forensics technologies to identify document forgery and signature forgery based on Blockchain technology.
If you’re looking for a document signing software that’s more secure than some of DocuSign alternative, such as government, legal, or healthcare, this is an excellent option. ZorroSign is also an environmentally responsible company as compared to other DocuSign Competitors.
➽ PDF Filler — Free DocuSign Competitor & Alternative
➽ CEO — Boris Shakhnovich
➽ Mobile App: iOS | Android
➽ Location —Brookline, Massachusetts
For editing, producing, signing, and maintaining PDF files online, pdfFiller is the best option available. It is one of the greatest alternative to DocuSign.
Using it since 2008, it has made it easier for companies and people to go paperless!
Additionally, pdfFiller is part of the airSlate Business Cloud, a simple bundle that offers a variety of useful services.
➽ Dochub — Free DocuSign Competitor & Alternative
➽ CEO — Chris Devor
➽ Mobile App: None, web-based only
➽ Location — Boston
Online PDF annotation and best electronic signing software, Dochub, allows users to add text and photos to their PDFs online. Multi-signer procedures, document signing in bulk, lossless editing, team collaboration, and more are all possible with this program.
The low cost of this program makes it a standout in comparison to DocuSign. The process of setting up and transmitting documents to users has been made simpler.
With DocHub’s editing features and the option to keep multiple signatures on several devices, it’s easy to collaborate with others. As a result of this functionality, it is a significant DocuSign competitor.
➽ EasySIGN— Free DocuSign Competitor & Alternative
➽ CEO — Not Found
➽ Mobile App: None, web-based only
➽ Location — Hapert, The Netherlands
They are proud to say that EasySIGN is acknowledged as a real one-stop-shop software solution for the signmaking and digital big format printing businesses worldwide.
The software provides unique, non-destructive design and production capabilities that are both inventive and easy to use in order to convert creative ideas into production-ready realities.
It offers a worldwide network of resellers that are experts in the field. More than 25 nations use the program, which has been translated into numerous languages and delivered to a dedicated clientele.
➽ Signority— Free DocuSign Competitor & Alternative
➽ CEO —Jane He
➽ Mobile App: None, web-based only
➽ Location — ON Canada
When you use Signority, the eSignature process is completely automated and your document management costs are much reduced. This allows you to focus more on your business.
Electronic signatures and reminders are both possible with the software, which delivers papers for digital signature and eSignature. All of your papers may be securely shared and kept in the cloud after they have been safely changed.
It enables process automation, corporate branding, real-time status alerts and traceability, and a host of additional benefits.
➽ Contract Book — Free DocuSign Competitor & Alternative
➽ CEO —Niels Martin Brochner
➽ Mobile App: None, web-based only
➽ Location — New York, USA
Contractbook is a collaborative contract management platform that automates your process and syncs contract data across your corporate platforms. It is also a good document signing software as well as DocuSign competitor.
You may use it to handle contracts effectively. All forms of legal papers may be signed, created, and stored digitally with this program. It also aids in the creation of a more open company environment.
Compliance is guaranteed and time is saved thanks to the software Using the solution, legal practitioners may monitor and manage their client’s contract online with ease and security.
➽ Proposify — Free DocuSign Competitor & Alternative
➽ CEO — Kyle Racki
➽ Mobile App: None, web-based only
➽ Location — Halifax, Nova Scotia
It’s a web-based proposal management system. By using Proposify, you can take charge of and gain insight into the most important portion of the sales process. Develop your self-assurance and flexibility to the point where you can dictate the terms of any business transaction.
Produce flawless and consistent sales materials. Get the info you need to expand your process, make appropriate commitments, and develop accurate forecasts. Streamline the approval procedure for your clients and new clients.
This competitor to DocuSign offers a wide variety of features, including an easy-to-use design editor, electronic signatures, CRM integration, data-driven insights, dynamic pricing and packaging, document management, approval workflows, and more.
➽ Qwilr — Free DocuSign Competitor & Alternative
➽ CEO — Dylan Baskind
➽ Mobile App: None, web-based only
➽ Location — Redfern, New South Wales
Qwilr intends to completely alter the online paper-making and distribution processes for corporations. With this innovation, companies can easily convert static web material into interactive, user-friendly mobile experiences. According to the vendor, this enables businesses to provide clients with quotes, proposals, and presentations while also taking advantage of analytics and other capabilities.
Increase your company’s sales output by freeing your sales staff from mundane duties like copying and pasting and allowing them to focus on more strategic initiatives. Build a database of documents that can be readily updated when you receive new contacts. Effective real-time collaboration is made simpler with intuitive commenting and discussion features.
Provide a quote that can be modified in real time, signed digitally, and used in other ways to meet the individual requirements of each client. Without leaving the Qwilr app, your clients can give final approval by checking boxes and providing comments.
Qwilr, like other DocuSign competitors, is a web-based document editor that could help your company save time and look more professional when communicating with customers.
➽ DealHub — Free DocuSign Competitor & Alternative
➽ CEO —Eyal Elbahary
➽ Mobile App: None, web-based only
➽ Location — Austin, Texas
The next notable DocuSign competitor is DealHub, which offers your business the most complete and unified revenue workflow available. Our zero-code platform is designed specifically to aid visionary leaders in improving cooperation, speeding up the sales cycle, and maintaining a steady flow of leads into the pipeline.
With the support of CPQ, CLM, and Subscription Management tools, which are all driven by an intuitive Sales Playbook, you can speed up the contract negotiation process, improve content delivery, and clinch more deals. A digital DealRoom is a platform where buyers and sellers can connect and exchange information in one convenient location.
Market leaders such as WalkMe, Gong, Drift, Hopin, Yotpo, Sendoso, and Braze are using DealHub to reduce their time to revenue and deliver a consistent sales experience for their sales teams and customers.
➽ DocSend— Free DocuSign Competitor & Alternative
➽ CEO — Russ Heddleston
➽ Mobile App: None, web-based only
➽ Location —San Francisco, CA
Sharing and managing the information that drives your business forward has never been easier than with Dropbox DocSend. Dropbox DocSend’s link-based approach makes it simple to tailor security to each recipient, track file views in real-time, evaluate content performance on a page-by-page basis, and establish up state-of-the-art virtual deal rooms. DocSend, like DocuSign’s competitors, is an e-signature service that may be used by any company.
The above-mentioned are some of the best DocuSign competitors that can help you to figure out which option suits your needs in the best ways. Many online signature software that have been discussed offer a free version so that the customers can find an easy and perfect solution.
If you ask our opinion then Signnow and WeSignature is the best option. It is the best one of the top online electronic signature software that come with a wide range of features. You can use this software to send unlimited documents while professionally designing documents for any purpose and numerous integrations.
You can begin a free trial and see how SignNow and WeSignature can help.
If you’re only searching for a way to submit and sign your electronic papers, DocuSign is fantastic. However, teams who wish to do more with their documents will find it to be a poor fit.
Here are some of the reasons why you might want to look at one of DocuSign’s competitors.
The market for electronic signatures is constantly changing.
Other significant e-signing hubs including SignNow, Wesignature, Signaturely, and PandaDoc are currently greatest DocuSign’s competitors.
On the market, there are several eSignature service providers, all of them provide slightly unique products. It can be difficult to decide which eSignature solutions will be most useful for you and which could be a waste of money, though, given the abundance of options. Prior to deciding on a DocuSign competitors for your company, consider the following points:
Finding the ideal DocuSign competitors for you will be much simpler now that you are aware of the answers to those concerns.
There are cheaper competitors to DocuSign, and SignNow is one of them. If your company needs an easy way to collect valid electronic signatures, look no further than SignNow eSignature software.
The excellent DocuSign competitor is SignNow. When you use SignNow to send out documents for signature, you may send a lot more of them at a much reduced cost. Use it more easily and get better customer service.
Subscribe to DDIntel Here.
Join our network here: https://datadriveninvestor.com/collaborate
—
—
1
empowerment through data, knowledge, and expertise. subscribe to DDIntel at https://ddintel.datadriveninvestor.com
AboutHelpTermsPrivacy
I’m interested by human creativity and technology. Nature enthusiast, self-motivator, visionary, and energetic communicator. Email me: tobykiernan1984@gmail.com
Help
Status
Writers
Blog
Careers
Privacy
Terms
About
Text to speech

source

Straight-up document scanning, basic connectivity
I focus on printer and scanner technology and reviews. I have been writing about computer technology since well before the advent of the internet. I have authored or co-authored 20 books—including titles in the popular Bible, Secrets, and For Dummies series—on digital design and desktop publishing software applications. My published expertise in those areas includes Adobe Acrobat, Adobe Photoshop, and QuarkXPress, as well as prepress imaging technology. (Over my long career, though, I have covered many aspects of IT.)
Brother's entry-level ADS-3100 is a capable sheetfed document scanner that's a good value for home, hybrid, or small offices and workgroups.
At the bottom of the pecking order in Brother’s recent release of a wave of sheetfed desktop document scanners, its ADS-3100 High-Speed Desktop Scanner ($329.99) is relatively fast and reliably accurate. It lists for about $40 less than the next model up, the ADS-3300W, but you give up quite a lot for the savings: network connectivity, a touch-screen control panel, and support for smartphones and other handheld devices, to name a few convenience and productivity features. That’s not to say, however, that small or home offices that plug the ADS-3100 into a solitary computer’s USB port (or scan directly to a USB flash, solid-state, or hard drive) won’t get good value from this scanner. Competition among entry-level and mid-volume document scanners is formidable, but if your business doesn’t demand networkability or wireless scanning, this Brother model should serve you well.
The ADS-3100 is the last of five new sheetfed document scanners from Brother to reach our test bench. The ADS-4900W is the high-volume flagship (and an Editors’ Choice award winner), and the midrange ADS-4700W, ADS-4300N, and ADS-3300W are excellent as well. They are all the same size. The five Brothers measure 7.5 by 11.7 by 8.5 inches (HWD) with their paper trays closed—like most scanners in this category, they double or triple their desktop footprint with trays extended for use—and weigh 6.2 to 6.5 pounds each.
This scanner has too many direct competitors to list here, so we’ll name just four key ones: the Fujitsu ScanSnap iX1400, the Epson WorkForce ES-400 II, the Raven Select Document Scanner, and the Canon imageFormula R50 Office Docment Scanner. The ADS-3100 falls short of some rivals by lacking a color touch screen. Its control panel holds only four buttons (Power, Stop/Cancel, Scan to USB, Scan to PC) and a few LED status indicators. This and the ADS-4300N are the only members of the new Brother quintet to lack touch screens.
You can scan to a variety of PDF types (high-compression, image, searchable, secure, or signed), as well as PDF/A, single-page and multipage TIFF, BMP, plain text, and Microsoft Word, Excel, and PowerPoint formats. The scanner’s maximum resolution is 600 dots per inch (1,200dpi interpolated), and it supports document sizes ranging from 2 inches square to 8.5 inches wide by 16.4 feet long, with 24-bit color depth. 
Online scanning destinations include cloud and social media sites and FTP sites, as well as local drives and email. It’s easy to access most social media and cloud sites, though the ADS-3100 is preconfigured to support Google Drive, OneDrive, Evernote, Box, Dropbox, OneNote, SharePoint Online, and Expensify.
The ADS-3100 features a 60-sheet automatic document feeder (ADF) for sending single- and double-sided multipage documents to the scanner. Brother rates its daily duty cycle at 6,000 scans. These specs match the ADS-4300N’s; the ADS-4700W has a larger 80-page ADF, while the more robust ADS-4900W combines a 100-sheet feeder and 9,000-scan daily duty cycle.
These specs are more or less average among low-end document scanners. The Raven Select, Epson ES-400 II, and ScanSnap iX1400 all have 50-sheet ADFs, while the Canon R50’s holds 60 sheets. The Fujitsu has the same 6,000-scan duty rating as the Brother; the Epson and Canon are rated for 4,000 and the Raven for only 2,000 daily scans.
Of the five new Brother document scanners, the ADS-3100 is the only one without a wired or wireless network interface—its sole connectivity option, apart from the USB Type-A port around back for scanning directly to storage devices, is a USB 3.0 (or 2.0) cable. That leaves out smartphones and tablets.
The software bundle doesn’t reflect this printer’s lower-end status, though. It includes Brother iPrint&Scan (desktop) for Windows and Mac, Brother ScanEssentials Lite for Windows, Kofax Power PDF for Windows, Presto! BizCard for Windows and Mac, Image Folio Processing Software for Windows, and Kofax PaperPort SE with OCR for Windows.
iPrint&Scan is an all-in-one printer interface that’s also compatible with Brother’s single-function scanners and printers. It lets you create and manage workflow profiles that you can choose with the front-panel buttons.
ScanEssentials Lite, meanwhile, is a trimmed-down version of another Brother scanner interface that’s also a document-management and financial-data archiving application. Kofax PaperPort also combines a scanner interface with document-management features, among them its own optical character recognition (OCR), workflow profiles, and automated naming conventions.
Presto! BizCard is what it sounds like, a business-card-scanning and contact-archiving program, and Image Folio is an image-capturing program designed to help you scan, edit, enhance, and print photos. You also get four third-party drivers—TWAIN, WIA, ISIS, and Sane—for scanning directly into many compatible applications, such as Adobe Acrobat and Photoshop and the Microsoft 365 suite.
Like its siblings (barring the flagship ADS-4900W), the ADS-3100 is rated at 40 one-sided (simplex) pages per minute (ppm) and 80 two-sided (duplex) images per minute (ipm, where each page side is counted as an image). I put those speed figures to the test using iPrint&Scan over a USB connection to our Intel Core i5 testbed running Windows 10 Pro. (I also ran a few tests with some of the other apps and got similar results, though scanning to USB flash drives is notably faster; see more about how we test scanners.)
First, I clocked the ADS-3100 as it scanned our standard one-sided 25-page and two-sided 25-page (50 sides) documents and then converted and saved them as image PDF files. The scanner managed 41.6ppm and 81.3ipm, barely beating its ratings. The competitors mentioned here did similarly, with the ADS-4900W being faster (68.7ppm and 125.4ipm) and the Epson ES-400 II being slower (37.7ppm and 66.7ipm).
Next, I timed the Brother as it scanned our two-sided, 25-page hard-copy document and saved it to the more versatile searchable PDF format. The ADS-3100 finished the job in 38 seconds, on the high side of average. The ADS-4900W and the HP ScanJet Pro N4000 snw1 took 34 seconds and 24 seconds respectively, while the Canon R50 took 37 seconds. The Fujitsu made it in 40 seconds, versus 41 for the Raven Select and 44 for the Epson. Frankly, unless you spend most of your day scanning stacks of lengthy documents, these scores should be fast enough for most offices.
Besides, how well a scanner reads text is more important than how quickly it does so. Most document scanners today can convert printed text to searchable PDF format with no errors down to 5 or 6 points. The Brother ADS-3100 was perfectly average, managing scans down to 6-point type problem-free in both our Arial and Times New Roman font tests. Of the other machines discussed here, only the Canon R50 yielded a different result: down to 5 points for Arial, and 6 points for Times New Roman. You’re not likely to encounter text smaller than 10 points in most real-world business documents.
I also scanned a few stacks of business cards into Presto! BizCard, with predictable results: The software does a fine job of digitizing text and figures and putting them into the proper fields in a contacts database (or exporting them to Outlook, Gmail, and other contact management or personal information manager apps). Finally, to see how well the Brother handled images, I scanned several photos to Image Folio. Most were well-detailed, with bright and accurate colors, showing the ADS-3100 can serve as a decent sheetfed photo scanner or alternative to the Epson FastFoto FF-680W and Canon imageFormula RS40. If you have a shoebox of photos stashed under the bed or on a closet shelf, the Image Folio software should deliver additional value.
If you can live without networked or mobile scanning, the Brother ADS-3100 may be right for you, especially if you also have a bunch of photos to archive (though it’s more of a document than a photo scanner). Without wireless or Ethernet connectivity, however, it’s really more of a personal machine for relatively low scanning volumes, around a few hundred per day. (You’d have to fill its 60-sheet ADF 100 times per day, or 14 times per hour, to reach its 6,000-scan limit.) Under the right conditions, though, the ADS-3100 is without question a fine entry-level-to-midrange document scanner.
Brother's entry-level ADS-3100 is a capable sheetfed document scanner that's a good value for home, hybrid, or small offices and workgroups.
Sign up for Lab Report to get the latest reviews and top product advice delivered right to your inbox.

This newsletter may contain advertising, deals, or affiliate links. Subscribing to a newsletter indicates your consent to our Terms of Use and Privacy Policy. You may unsubscribe from the newsletters at any time.
Your subscription has been confirmed. Keep an eye on your inbox!
Advertisement
I focus on printer and scanner technology and reviews. I have been writing about computer technology since well before the advent of the internet. I have authored or co-authored 20 books—including titles in the popular Bible, Secrets, and For Dummies series—on digital design and desktop publishing software applications. My published expertise in those areas includes Adobe Acrobat, Adobe Photoshop, and QuarkXPress, as well as prepress imaging technology. (Over my long career, though, I have covered many aspects of IT.)
In addition to writing hundreds of articles for PCMag, over the years I have also written for many other computer and business publications, among them Computer Shopper, Digital Trends, MacUser, PC World, The Wirecutter, and Windows Magazine. I also served as the Printers and Scanners Expert at About.com (now Lifewire).
Read William’s full bio
Advertisement
PCMag.com is a leading authority on technology, delivering lab-based, independent reviews of the latest products and services. Our expert industry analysis and practical solutions help you make better buying decisions and get more from technology.
PCMag supports Group Black and its mission to increase greater diversity in media voices and media ownerships.
© 1996-2022 Ziff Davis, LLC., a Ziff Davis company. All Rights Reserved.
PCMag, PCMag.com and PC Magazine are among the federally registered trademarks of Ziff Davis and may not be used by third parties without explicit permission. The display of third-party trademarks and trade names on this site does not necessarily indicate any affiliation or the endorsement of PCMag. If you click an affiliate link and buy a product or service, we may be paid a fee by that merchant.

source

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.
Advertisement
Carousel with three slides shown at a time. Use the Previous and Next buttons to navigate three slides at a time, or the slide dot buttons at the end to jump three slides at a time.
24 May 2022
Matej Petković, Luke Lucas, … Dragi Kocev
22 May 2020
Miguel Luengo-Oroz, Katherine Hoffmann Pham, … Bernardo Mariano
13 April 2020
Julia Stoyanovich, Jay J. Van Bavel & Tessa V. West
20 January 2022
Tirtharaj Dash, Sharad Chitlangia, … Ashwin Srinivasan
07 May 2019
A. Gonoskov, E. Wallin, … I. Meyerov
06 November 2020
Jayaraman J. Thiagarajan, Bindya Venkatesh, … Brian Spears
16 July 2021
Ken Hasselmann, Antoine Ligot, … Mauro Birattari
03 February 2020
Rafael A. Calvo, Dorian Peters & Stephen Cave
18 July 2018
Jarosław M. Granda, Liva Donina, … Leroy Cronin
Nature Communications volume 13, Article number: 6039 (2022) Cite this article
14k Accesses
54 Altmetric
Metrics details
The development and deployment of machine learning systems can be executed easily with modern tools, but the process is typically rushed and means-to-an-end. Lack of diligence can lead to technical debt, scope creep and misaligned objectives, model misuse and failures, and expensive consequences. Engineering systems, on the other hand, follow well-defined processes and testing standards to streamline development for high-quality, reliable results. The extreme is spacecraft systems, with mission critical measures and robustness throughout the process. Drawing on experience in both spacecraft engineering and machine learning (research through product across domain areas), we’ve developed a proven systems engineering approach for machine learning and artificial intelligence: the Machine Learning Technology Readiness Levels framework defines a principled process to ensure robust, reliable, and responsible systems while being streamlined for machine learning workflows, including key distinctions from traditional software engineering, and a lingua franca for people across teams and organizations to work collaboratively on machine learning and artificial intelligence technologies. Here we describe the framework and elucidate with use-cases from physics research to computer vision apps to medical diagnostics.
The accelerating use of artificial intelligence (AI) and machine learning (ML) technologies in systems of software, hardware, data, and people introduce vulnerabilities and risks due to dynamic and unreliable behaviors; fundamentally, ML systems learn from data, introducing known and unknown challenges in how these systems behave and interact with their environment. Currently, the approach to building AI technologies is siloed: models and algorithms are developed in testbeds isolated from real-world environments, and without the context of larger systems or broader products they will be integrated within for deployment. The main concern is models are typically trained and tested on only a handful of curated datasets, without measures and safeguards for future scenarios, and oblivious of the downstream tasks and users. Even more, models and algorithms are often integrated into a software stack without regard for the inherent stochasticity and failure modes of the hidden ML components. Consider the massive effect random seeds have on deep reinforcement learning model performance¹, for instance.
Other domains of engineering, such as civil and aerospace, follow well-defined processes and testing standards to streamline development for high-quality, reliable results. Technology Readiness Level (TRL) is a systems engineering protocol for deep tech² and scientific endeavors at scale, ideal for integrating many interdependent components and cross-functional teams of people. It is no surprise that TRL is a standard process and parlance in NASA³ and DARPA⁴.
For a spaceflight project, there are several defined phases, from pre-concept to prototyping to deployed operations to end-of-life, each with a series of exacting development cycles and reviews. This is in stark contrast to common machine learning and software workflows, which promote quick iteration, rapid deployment, and simple linear progressions. Yet the NASA technology readiness process for spacecraft systems is overkill; we need robust ML technologies integrated with larger systems of software, hardware, data, and humans, but not necessarily for missions to Mars. We aim to bring systems engineering to AI and ML by defining and putting into action a lean Machine Learning Technology Readiness Levels (MLTRL) framework. We draw on decades of AI and ML development, from research through production, across domains and diverse data scenarios: for example, computer vision in medical diagnostics and consumer apps, automation in self-driving vehicles and factory robotics, tools for scientific discovery and causal inference, streaming time-series in predictive maintenance and finance.
In this paper, we define our framework for developing and deploying robust, reliable, and responsible ML and data systems, with several real test cases of advancing models and algorithms from R&D through productization and deployment, including essential data considerations—Fig. 1 illustrates the overall MLTRL process. Additionally, MLTRL prioritizes the role of AI ethics and fairness, and our systems AI approach can help curb the large societal issues that can result from poorly deployed and maintained AI and ML technologies, such as the automation of systemic human bias, denial of individual autonomy, and unjustifiable outcomes (see the Alan Turing Institute Report on Ethical AI⁵). The adoption and proliferation of MLTRL provide a common nomenclature and metric across teams and industries. The standardization of MLTRL across the AI industry should help teams and organizations develop principled, safe, and trusted technologies.
Most ML workflows prescribe an isolated, linear process of data processing, training, testing, and serving a model³⁷. Those workflows fail to define how ML development must iterate over that basic process to become more mature and robust, and how to integrate with a much larger system of software, hardware, data, and people. Not to mention MLTRL continues beyond deployment: monitoring and feedback cycles are important for continuous reliability and improvement over the product lifetime.
MLTRL defines technology readiness levels (TRLs) to guide and communicate machine learning and artificial intelligence (ML/AI) development and deployment. A TRL represents the maturity of a model or algorithm, data pipelines, software module, or composition thereof; a typical ML system consists of many interconnected subsystems and components, and the TRL of the system is the lowest level of its constituent parts⁶. Note we use “model” and “algorithm” somewhat interchangeably when referring to the technology under development. The same MLTRL process and methods apply for a machine translation model and for an A/B testing algorithm, for example. The anatomy of a level is marked by gated reviews, evolving working groups, requirements documentation with risk calculations, progressive code and testing standards, and deliverables such as TRL Cards (Fig. 2) and ethics checklists. Templates and examples for MLTRL deliverables will be open-sourced upon publication at ai-infrastructure.org/mltrl. These components—which are crucial for implementing the levels in a systematic fashion—as well as MLTRL metrics and methods are concretely described in examples and in the “Methods” section. Lastly, to emphasize the importance of data tasks in ML, from data curation⁷ to data governance⁸, we state several important data considerations at each MLTRL level.
Here is an example reflecting a neuropathology machine vision use-case²², detailed in the “Discussion” section. Note this is a subset of a full TRL Card, which in reality lives as a full document in an internal wiki. Notice the card clearly communicates the data sources, versions, and assumptions. This helps mitigate invalid assumptions about performance and generalizability when moving from R&D to production and promotes the use of real-world data earlier in the project lifecycle. We recommend documenting datasets thoroughly with semantic versioning and tools such as datasheets for datasets⁷⁶, and following data accountability best practices as they evolve (see ref. 81).
The levels are briefly defined as follows and in Fig. 1, and elucidated with real-world examples later.
This is a stage for greenfield AI research, initiated with a novel idea, guiding question, or poking at a problem from new angles. The work mainly consists of literature review, building mathematical foundations, white-boarding concepts, and algorithms, and building an understanding of the data—for work in theoretical AI and ML, however, there will not yet be data to work with (for example, a novel algorithm for Bayesian optimization⁹, which could eventually be used for many domains and datasets). The outcome of Level 0 is a set of concrete ideas with sound mathematical formulation, to pursue through low-level experimentation in the next stage. When relevant, this level expects conclusions about data readiness, including strategies for getting the data to be suitable for the specific ML task. To graduate, the basic principles, hypotheses, data readiness, and research plans need to be stated, referencing relevant literature. With graduation, a TRL Card should be started to succinctly document the methods and insights thus far—this key MLTRL deliverable is detailed in the “Methods” section and Fig. 2.
Level 0 data—Not a hard requirement at this stage as it is largely relevant to theoretical machine learning. That being said, data availability needs to be considered for defining any research project to move past theory.
Level 0 review—The reviewer here is solely the lead of the research lab or team, for instance, a Ph.D. supervisor. We assess hypotheses and explorations for mathematical validity and potential novelty or utility, not necessarily code nor end-to-end experiment results.
To progress from basic principles to practical use, we design and run low-level experiments to analyze specific model or algorithm properties (rather than end-to-end runs for a performance benchmark score). This involves the collection and processing of sample data to train and evaluate the model. This sample data need not be the full data; it may be a smaller sample that is currently available or more convenient to collect. In some cases it may suffice to use synthetic data as the representative sample—in the medical domain, for example, acquiring datasets can take many months due to security and privacy constraints, so generating sample data can mitigate this blocker from early ML development. Further, working with the sample data provides a blueprint for the data collection and processing pipeline (including answering whether it is even possible to collect all necessary data), that can be scaled up for the next steps. The experiments, good results or not, and mathematical foundations need to pass a review process with fellow researchers before graduating to Level 2. The application is still speculative, but through comparison studies and analyses, we start to understand if/how/where the technology offers potential improvements and utility. Code is research-caliber: The aim here is to be quick and dirty, moving fast through iterations of experiments. Hacky code is okay, and full test coverage is actually discouraged, as long as the overall codebase is organized and maintainable. It is important to start semantic versioning practices early in the project lifecycle, which should cover code, models, and datasets. This is crucial for retrospectives and reproducibility, issues that can be costly and severe at later stages. This versioning information and additional progress should be reported on the TRL Card (see, for example, Fig. 2).
Level 1 data—At minimum, we work with sample data that is representative of downstream real datasets, which can be a subset of real data, synthetic data, or both. Beyond driving low-level ML experiments, the sample data forces us to consider data acquisition and processing strategies at an early stage before it becomes a blocker later.
Level 1 review—The panel for this gated review is entirely members of the research team, reviewing for scientific rigor in early experimentation, and pointing to important concepts and prior work from their respective areas of expertise. There may be several iterations of feedback and additional experiments.
Active R&D is initiated, mainly by developing and running in testbeds: simulated environments and/or simulated data that closely matches the conditions and data of real scenarios—note these are driven by model-specific technical goals, not necessarily application or product goals (yet). An important deliverable at this stage is the formal research requirements document (with well-specified verification and validation (V&V¹⁰) steps). A requirement is a singular documented physical or functional need that a particular design, product, or process aims to satisfy. Requirements aim to specify all stakeholders’ needs while not specifying a specific solution. Definitions are incomplete without corresponding measures for verification and validation (V&V). Verification: Are we building the product right? Validation: Are we building the right product?¹⁰ Here is one of several key decision points in the broader process: The R&D team considers several paths forward and sets the course: (A) prototype development towards Level 3, (B) continued R&D for longer-term research initiatives and/or publications, or some combination of A and B. We find the culmination of this stage is often a bifurcation: some work moves to applied ML, while some circles back for more research. This common MLTRL cycle is an instance of the non-monotonic discovery switchback mechanism (detailed in the “Methods” section and Fig. 3).
The difference is the former is circumstantial while the latter is predefined in the process. The other embedded switchback we define in the main MLTRL process is from level 9 to 4, shown in Fig. 4. While it is true that the majority of ML projects start at a reasonable readiness out of the box, e.g. level 4, this can make it challenging and problematic to switchback to R& D levels that the team have not encountered and may not be equipped for. In the right diagram we show a common review switchback from Level 5 to 4 (staying in the prototyping phase (orange)), and a switchback (faded) that should not be implemented because the prior level was not explicitly done; level 2 is squarely in the research pipeline (red).
Level 2 data—Datasets at this stage may include publicly available benchmark datasets, semi-simulated data based on the data sample in Level 1, or fully simulated data based on certain assumptions about the potential deployment environments. The data should allow researchers to characterize model properties, and highlight corner cases or boundary conditions, in order to justify the utility of continuing R&D on the model.
Level 2 review—To graduate from the PoP stage, the technology needs to satisfy research claims made in previous stages (brought to bear by the aforementioned PoP data in both quantitative and qualitative ways) with the analyses well-documented and reproducible.
Here we have checkpoints that push code development towards interoperability, reliability, maintainability, extensibility, and scalability. Code becomes prototype-caliber: A significant step up from research code in robustness and cleanliness. This needs to be well-designed, well-architected for dataflow and interfaces, generally covered by unit and integration tests, meet team style standards, and sufficiently documented. Note the programmers’ mentality remains that this code will someday be refactored/scrapped for productization; prototype code is relatively primitive with regard to the efficiency and reliability of the eventual system. With the transition to Level 4 and proof-of-concept mode, the working group should evolve to include product engineering to help define service-level agreements and objectives (SLAs and SLOs) of the eventual production system.
Level 3 data—For the most part consistent with Level 2; in general, the previous level review can elucidate potential gaps in data coverage and robustness to be addressed in the subsequent level. However, for test suites developed at this stage, it is useful to define dedicated subsets of the experiment data as default testing sources, as well as set up mock data for specific functionalities and scenarios to be tested.
Level 3 review—Teammates from applied AI and engineering are brought into the review to focus on sound software practices, interfaces and documentation for future development, and version control for models and datasets. There are likely domain- or organization-specific data management considerations going forward that this review should point out—e.g. standards for data tracking and compliance in healthcare¹¹.
This stage is the seed of application-driven development; for many organizations this is the first touch-point with product managers and stakeholders beyond the R&D group. Thus TRL Cards and requirements documentation are instrumental in communicating the project status and onboarding new people. The aim is to demonstrate the technology in a real scenario: quick proof-of-concept examples are developed to explore candidate application areas and communicate the quantitative and qualitative results. It is essential to use real and representative data for these potential applications. Thus data engineering for the PoC largely involves scaling up the data collection and processing from Level 1, which may include collecting new data or processing all available data using scaled experiment pipelines from Level 3. In some scenarios, there will be new datasets brought in for the PoC, for example, from an external research partner as a means of validation. Hand-in-hand with the evolution from sample to real data, the experiment metrics should evolve from ML research to the applied setting: proof-of-concept evaluations should quantify model and algorithm performance (e.g., precision and recall and various data splits), computational costs (e.g., CPU vs. GPU runtimes), and also metrics that are more relevant to the eventual end-user (e.g., number of false positives in the top-N predictions of a recommender system). We find this PoC exploration reveals specific differences between clean and controlled research data versus noisy and stochastic real-world data. The issues can be readily identified because of the well-defined distinctions between those development stages in MLTRL, and then targeted for further development.
AI ethics processes vary across organizations, but all should engage in ethics conversations at this stage, including ethics of data collection, and the potential of any harm or discriminatory impacts due to the model (as the AI capabilities and datasets are known). MLTRL requires ethics considerations to be reported on TRL Cards at all stages, which generally link to an extended ethics checklist. The key decision point here is to push onward with application development or not. It is common to pause projects that pass Level 4 review, waiting for a better time to dedicate resources, and/or pull the technology into a different project.
Level 4 data—Unlike the previous stages, having real-world and representative data is critical for the PoC; even with methods for verifying that data distributions in synthetic data reliably mirror those of real data, sufficient confidence in the technology must be achieved with real-world data of the use-case. Further, one must consider how to obtain high-quality and consistent data required for the future model inference: generation of the data pipeline PoC that will resemble the future inference pipeline that will take data from intended sources, transform it into features, and send it to the model for inference.
Level 4 review—Demonstrate the utility towards one or more practical applications (each with multiple datasets), taking care to communicate assumptions and limitations, and again reviewing data-readiness: evaluating the real-world data for quality, validity, and availability. The review also evaluates security and privacy considerations—defining these in the requirements document with risk quantification is a useful mechanism for mitigating potential issues (discussed further in the Methods section).
At this stage the technology is more than an isolated model or algorithm, it is a specific capability. For instance, producing depth images from stereo vision sensors on a mobile robot is a real-world capability beyond the isolated ML technique of self-supervised learning for RGB stereo disparity estimation. In many organizations, this represents a technology transition or handoff from R&D to productization. MLTRL makes this transition explicit, evolving the requisite work, guiding documentation, objectives and metrics, and team; indeed, without MLTRL it is common for this stage to be erroneously leaped completely, as shown in Fig. 4. An interdisciplinary working group is defined, as we start developing the technology in the context of a larger real-world process—i.e., transitioning the model or algorithm from an isolated solution to a module of a larger application. Just as the ML technology should no longer be owned entirely by ML experts, steps have been taken to share the technology with others in the organization via demos, example scripts, and/or an API; the knowledge and expertise cannot remain within the R&D team, let alone an individual ML developer. Graduation from Level 5 should be difficult, as it signifies the dedication of resources to push this ML technology through productization. This transition is a common challenge in deep-tech, sometimes referred to as “the valley of death” because project managers and decision-makers struggle to allocate resources and align technology roadmaps to effectively move to Levels 6, 7, and onward. MLTRL directly addresses this challenge by stepping through the technology transition or handoff explicitly.
In the left diagram (a subset of the Fig. 1 pipeline, same colors), the arrows show a common development pattern with MLTRL in the industry): projects go back to the ML toolbox to develop new features (dashed line), and frequent, incremental improvements are often a practice of jumping back a couple of levels to Level 7 (which is the main systems integrations stage). At Levels 7 and 8 we stress the need for tests that run use-case-specific critical scenarios and data-slices, which are highlighted by a proper risk-quantification matrix⁷⁸. Reviews at these Levels commonly catch gaps or oversight in the test and validation scenarios, resulting in frequent cycles back to Level 7 from 8. Cycling back to previous lower levels is not just a late-stage mechanism in MLTRL, but rather “switchbacks” occur throughout the process. Cycling back to Level 7 from 8 for more tests is an example of a review switchback, while the solid line from Level 9 to 7 is an embedded switchback where MLTRL defines certain conditions that require cycling back levels—see more in the “Methods” section and throughout the text. In the right diagram, we show the more common approach in the industry (without using our framework), which skips essential technology transition stages (gray)—ML Engineers push straight through to deployment, ignoring important productization and systems integration factors. This will be discussed in more detail in the “Methods” section.
Level 5 data—For the most part consistent with Level 4. However, considerations need to be taken for scaling of data pipelines: there will soon be more engineers accessing the existing data and adding more, and the data will be getting much more use, including automated testing in later levels. With this scaling can come challenges with data governance. The data pipelines likely do not mirror the structure of the teams or broader organization. This can result in data silos, duplications, unclear responsibilities, and missing control of data over its entire lifecycle. These challenges and several approaches to data governance (planning and control, organizational, and risk-based) are detailed in Janssen et al.⁸.
Level 5 review—The verification and validation (V&V) measures and steps defined in earlier R&D stages (namely Level 2) must all be completed by now, and the product-driven requirements (and corresponding V&V) are drafted at this stage. We thoroughly review them here and make sure there is stakeholder alignment (at the first possible step of productization, well ahead of deployment).
The main work here is significant software engineering to bring the code up to product-caliber: This code will be deployed to users and thus needs to follow precise specifications, have comprehensive test coverage, well-defined APIs, etc. The resulting ML modules should be robustified towards one or more target use-cases. If those target use-cases call for model explanations, the methods need to be built and validated alongside the ML model, and tested for their efficacy in faithfully interpreting the model’s decisions—crucially, this needs to be in the context of downstream tasks and the end-users, as there is often a gap between ML explainability that serves ML engineers rather than external stakeholders¹². Similarly, we need to develop the ML modules with known data challenges in mind, specifically to check the robustness of the model (and broader pipeline) to changes in the data distribution between development and deployment.
The deployment setting(s) should be addressed thoroughly in the product requirements document, as ML serving (or deploying) is an overloaded term that needs careful consideration. First, there are two main types: internal, as APIs for experiments and other usages mainly by data science and ML teams, and external, meaning an ML model that is embedded or consumed within a real application with real users. The serving constraints vary significantly when considering cloud deployment vs on-premise or hybrid, batch or streaming, open-source solution or containerized executable, etc. Even more, the data at deployment may be limited due to compliance, or we may only have access to encrypted data sources, some of which may only be accessible locally—these scenarios may call for advanced ML approaches such as federated learning¹³ and other privacy-oriented ML¹⁴. And depending on the application, an ML model may not be deployable without restrictions; this typically means being embedded in a rules engine workflow where the ML model acts like an advisor that discovers edge cases in rules. These deployment factors are hardly considered in model and algorithm development despite their significant influence on modeling and algorithmic choices; that said, hardware choices typically are considered early on, such as GPU versus edge devices. It is crucial to make these systems decisions at Level 6—not too early that serving scenarios and requirements are uncertain, and not too late that corresponding changes to model or application development risk deployment delays or failures. This marks a key decision for the project lifecycle, as this expensive ML deployment risk is common without MLTRL (see Fig. 4).
Level 6 data—Additional data should be collected and operationalized at this stage towards robustifying the ML models, algorithms, and surrounding components. These include adversarial examples to check local robustness¹⁵, semantically equivalent perturbations to check the consistency of the model with respect to domain assumptions^16,17, and collecting data from different sources and checking how well the trained model generalizes to them. These considerations are even more vital in the challenging deployment domains mentioned above with limited data access.
Level 6 review—Focus is on the code quality, the set of newly defined product requirements, system SLA and SLO requirements, data pipelines spec, and an AI ethics revisit now that we are closer to a real-world use-case. In particular, regulatory compliance is mandated for this gated review; the data privacy and security laws are changing rapidly, and missteps with compliance can make or break the project.
For integrating the technology into existing production systems, we recommend the working group has a balance of infrastructure engineers and applied AI engineers—this stage of development is vulnerable to latent model assumptions and failure modes, and as such cannot be safely developed solely by software engineers. Important tools for them to build together include:
Tests that run use-case-specific critical scenarios and data-slices—a proper risk-quantification table will highlight these.
A “golden dataset” should be defined to baseline the performance of each model and succession of models—see the computer vision app example in Fig. 5—for use in the continuous integration and deployment (CI/CD) tests.
Complicated logic such as this can mask ML model performance lags and failures, and also emphasized the need for R& D-to-product handoff described in MLTRL. Additional emphasis is placed on ML tests that consider the mix of real-world data with user annotations (b, right) and synthetic data generated by Unity AI’s Perception tool and structured domain randomization (b, left).
Metamorphic testing: A software engineering methodology for testing a specific set of relations between the outputs of multiple inputs. When integrating ML modules into larger systems, a codified list of metamorphic relations¹⁸ can provide valuable verification and validation measures and steps.
Data intervention tests that seek data bugs at various points in the pipelines, downstream to measure the potential effects of data processing and ML on consumers or users of that data, as well as upstream at data ingestion or creation. Rather than using model performance as a proxy for data quality, it is crucial to use intervention tests that instead catch data errors with mechanisms specific to data validation.
These tests in particular help mitigate underspecification in ML pipelines, a key obstacle to reliably training models that behave as expected in deployment¹⁹. On the note of reliability, it is important that quality assurance engineers (QA) play a key role here and through Level 9, overseeing data processes to ensure privacy and security, and covering audits for downstream accountability of AI methods.
Level 7 data—In addition to the data for test suites discussed above, this level calls for QA to prioritize data governance: how data is obtained, managed, used, and secured by the organization. This was earlier suggested in level 5 (in order to preempt related technical debt) and is essential here at the main junction for integration, which may create additional governance challenges in light of downstream effects and consumers.
Level 7 review—The review should focus on the data pipelines and test suites; a scorecard like the ML Testing Rubric²⁰ is useful. The group should also emphasize ethical considerations at this stage, as they may be more adequately addressed now (where there are many test suites put into place) rather than close to shipping later.
The technology is demonstrated to work in its final form and under expected conditions. There should be additional tests implemented at this stage covering deployment aspects, notably A/B tests, blue/green deployment tests, shadow testing, and canary testing, which enables proactive and gradual testing for changing ML methods and data. Ahead of deployment, the CI/CD system should be ready to regularly stress test the overall system and ML components. In practice, problems stemming from real-world data are impossible to anticipate and design for—an upstream data provider could change formats unexpectedly or a physical event could cause the customer behavior to change. Running models in shadow mode for a period of time would help stress test the infrastructure and evaluate how susceptible the ML model(s) will be to performance regressions caused by data. We observe that ML systems with data-oriented architectures are more readily tested in this manner, and better surface data quality issues, data drifts, and concept drifts—this is discussed later in the Beyond Software Engineering section. To close this stage, the key decision is go or no-go for deployment, and when.
Level 8 data—If not already in place, there absolutely needs to be mechanisms for automatically logging data distributions alongside model performance once deployed.
Level 8 review—A diligent walkthrough of every technical and product requirement, showing the corresponding validations, and the review panel is representative of the full slate of stakeholders.
In deploying AI and ML technologies, there is a significant need to monitor the current version and explicit considerations towards improving the next version. For instance, performance degradation can be hidden and critical, and feature improvements often bring unintended consequences and constraints. Thus at this level, the focus is on maintenance engineering—i.e., methods and pipelines for ML monitoring and updating. Monitoring for data quality, concept drift, and data drift is crucial; no AI system without thorough tests for these can reliably be deployed. By the same token, there must be automated evaluation and reporting—if actuals²¹ are available, continuous evaluation should be enabled, but in many cases, actuals come with a delay, so it is essential to record model outputs to allow for efficient evaluation after the fact. To these ends, the ML pipeline should be instrumented to log system metadata, model metadata, and data itself.
Monitoring for data quality issues and data drifts is crucial to catch deviations in model behavior, particularly those that are non-obvious in the model or product end-performance. Data logging is unique in the context of ML systems: data logs should capture statistical properties of input features and model predictions, and capture their anomalies. With monitoring for data, concept, and model drifts, the logs are to be sent to the relevant systems, applied, and research engineers. The latter is often non-trivial, as the model server is not ideal for model “observability” because it does not necessarily have the right data points to link the complex layers needed to analyze and debug models. To this end, MLTRL requires the drift tests to be implemented at stages well ahead of deployment, earlier than standard practice. Again we advocate for data-first architectures rather than the software industry-standard design by services (discussed later), which aids in surfacing and logging the relevant data types and slices when monitoring AI systems. For retraining and improving models, monitoring must be enabled to catch training-serving skew and let the team know when to retrain. Towards model improvements, adding or modifying features can often have unintended consequences, such as introducing latencies or even bias. To mitigate these risks, MLTRL has an embedded switchback here: any component or module changes to the deployed version must cycle back to Level 7 (integrations stage) or earlier—see Fig. 4. Additionally, for quality ML products, we stress a defined communication path for user feedback without roadblocks to R&D; we encourage real-world feedback all the way to research, providing valuable problem constraints and perspectives.
Level 9 data—Proper mechanisms for logging and inspecting data (alongside models) is critical for deploying reliable AI and ML—systems that learn on data have unique monitoring requirements (detailed above). In addition to the infrastructure and test suites covering data and environment shifts, it’s important for product managers and other owners to be on top of data policy shifts in domains such as finance and healthcare.
Level 9 review—The review at this stage is unique, as it also helps in lifecycle management: at a regular cadence that depends on the deployed system and domain of use, owners and other stakeholders are to revisit this review and recommend switchbacks if needed (discussed in the Methods section). This additional oversight at deployment is shown to help define regimented release cycles of updated versions, and provide another “eye” check for stale model performance or other system abnormalities.
Notice MLTRL is defined as stages or levels, yet much of the value in practice is realized in the transitions: MLTRL enables teams to move from one level to the next reliably and efficiently and provides a guide for how teams and objectives evolve with the progressing technology.
MLTRL is designed to apply to many real-world use-cases involving data and ML, from simple regression models used for predictive modeling energy demand or anomaly detection in data centers, to real-time modeling in rideshare applications and motion planning in warehouse robotics. For simple use-cases MLTRL may be overkill, and a subset may suffice—for instance, model cards as demonstrated by Google for basic image classification. Yet this is a fine line, as the same cards-only approach in the popular “Huggingface” codebases is too simplistic for the language models they represent, deployed in domains that carry significant consequences. MLTRL becomes more valuable with more complex, larger systems and environments, especially in risk-averse domains.
In this section, we illustrate MLTRL in several real use-cases from a diverse array of domains. In each use-case, we first outline the specific challenges faced in that domain, then move on to demonstrate how these challenges are addressed in the MLTRL framework—highlighting the specific levels that deal with each challenge. Moreover, for each use-case we provide a step-by-step, level-by-level walk-through of how MLTRL is applied, thus outlining in concrete, real-world settings how the MLTRL framework should be utilized.
While most ML projects begin with a specific task and/or dataset, there are many that originate in ML theory without any target application—i.e., projects starting MLTRL at level 0 or 1. These projects nicely demonstrate the utility of MLTRL built-in switchbacks, bifurcating paths, and iteration with domain experts. An example we discuss here is a novel approach to representing data in generative vision models from Naud and Lavin²², which was then developed into state-of-the-art unsupervised anomaly detection, and targeted for two human-machine visual inspection applications: First, industrial anomaly detection, notably in precision manufacturing, to identify potential errors for human-expert manual inspection. Second, using the model to improve the accuracy and efficiency of neuropathology, the microscopic examination of neurosurgical specimens for cancerous tissue. In these human-machine teaming use-cases there are specific challenges impeding practical, reliable use:
Hidden feedback loops can be common and problematic in real-world systems influencing their own training data: over time the behavior of users may evolve to select data inputs they prefer for the specific AI system, representing some skew from the training data. In this neuropathology case, selecting whole-slide images that are uniquely difficult for manual inspection, or even biased by that individual user. Similarly, we see underlying healthcare processes can act as hidden confounders, resulting in unreliable decision support tools²³.
Model availability can be limited in many deployment settings: for example, on-premises deployments (common in privacy-preserving domains like healthcare and banking), edge deployments (common in industrial use-cases such as manufacturing and agriculture), or from the infrastructure’s inability to scale to the volume of requests. This can severely limit the team’s ability to monitor, debug, and improve deployed models.
Uncertainty estimation is valuable in many AI scenarios, yet not straightforward to implement in practice. This is further complicated with multiple data sources and users, each injecting generally unknown amounts of noise and uncertainties. In medical applications it is of critical importance, to provide measures of confidence and sensitivity, and for AI researchers through end-users. In anomaly detection, various uncertainty measures can help calibrate the false-positive versus false-negative rates, which can be very domain specific.
Costs of edge cases can be significant, sometimes risking expensive machine downtime or medical failures. This is exacerbated in anomaly detection anomalies are by definition rare so they can be difficult to train for, especially for the anomalies that are completely unseen until they arise in the wild.
End-user trust can be difficult to achieve, often preventing the adoption of ML applications, particularly in the healthcare domain and other highly regulated industries.
These and additional ML challenges such as data privacy and interpretability can inhibit ML adoption in clinical practice and industrial settings but can be mitigated with MLTRL processes. We’ll describe how in the context of the ref. 22 example, which began at level 0 with theoretical ML work on manifold geometries, and at level 5 was directed towards specialized human-machine teaming applications utilizing the same ML method under-the-hood.
Levels 0–1—From open-ended exploration of data-representation properties in various Riemmanian manifold curvatures, we derived from first principles and empirically identified a property with hyperbolic manifolds: when used as a latent space for embedding data without labels, the geometry organizes the data by its implicit hierarchical structure. Unsupervised computer vision was identified in reviews as a promising direction for proof-of-principle work.
Level 2—One approach for validating the earlier theoretical developments was to generate synthetic data to isolate very specific features in data we would expect represented in the latent manifold. The results showed promise for anomaly detection—using the latent representation of data to automatically identify images that are out-of-the-ordinary (anomalous), and also using the manifold to inspect how they are semantically different. Further, starting with an implicitly probabilistic modeling approach implied uncertainty estimation could be a valuable feature downstream. This made the level 2 key decision point clear: proceed with applied ML development.
Levels 3–5—Proof-of-concept development and reviews demonstrated promise for several commercial applications relevant to the business, and also highlighted the need for several key features (defined as R&D and product requirements): interpretability (towards end-user trust), uncertainty quantification (to show confidence scores), and human-in-the-loop (for domain expertise). Without the MLTRL PoC steps and review processes, these features can often be delayed until beta testing or overlooked completely—for example, the failures of applying IBM Watson in medical applications²⁴. For this technology, the applications to develop towards are anomaly detection in histopathology and manufacturing, specifically inspecting whole-slide images of neural tissue, and detecting defects in metallic surfaces, respectively. From the systems perspective, we suggest quantifying the uncertainties of components and propagating them through the system, which can improve safety and trust. Probabilistic ML methods, rooted in Bayesian probability theory, provide a principled approach to representing and manipulating uncertainty about models and predictions²⁵. For this reason, we advocate strongly for probabilistic models and algorithms in AI systems. In this machine vision example, the MLTRL technical requirements specifically called for a probabilistic generative model to readily quantify various types of uncertainties and propagate them forward to the visualization component of the pipeline, and the product requirements called for the downstream confidence and sensitivity measures to be exposed to the end-user. Component uncertainties must be assembled in a principled way to yield a meaningful measure of overall system uncertainty, based on which safe decisions can be made²⁶. See the “Methods” section for more on uncertainty in AI systems. The early checks for data management and governance proved valuable here, as the application areas dealt with highly sensitive data that would significantly influence the design of data pipelines and test suites. In both the neuropathology and manufacturing applications, the data management checks also raised concerns about hidden feedback loops, where users may unintentionally skew the data inputs when using the anomaly detection models in practice, for instance biasing the data towards specific subsets they subjectively need help with. Incorporating domain experts this early in the project lifecycle helped inform verification and validation steps to help be robust to the hidden feedback loops. Not to mention their input guided us towards user-centric metrics for performance, which can often skew from ML metrics in important ways—for instance, the typical acceptance ratio for false positives versus false negatives does not apply to select edge cases, for which our hierarchical anomaly classification scheme was useful²². From prior reviews and TRL card documentation, we also identified the value of synthetic data generation in application development: anomalies are by definition rare so they are hard to come by in real datasets, especially with evolving environments in deployment settings, so the ability to generate synthetic datasets for anomaly detection can accelerate the levels 6–9 pipeline, and help ensure more reliable models in the wild.
Level 6 (medical)—The medical inspection application experienced a bifurcation with product work proceeding while additional R&D was desired to explore improved data processing methods while engaging with clinicians and medical researchers for feedback. Proceeding through the levels in a non-linear, non-monotonic way is common in MLTRL and encouraged by various switchback mechanisms (detailed in the “Methods” section). These practices—intentional switchbacks, frequent engagement with domain experts and users—can help mitigate methodological flaws and underlying biases that are common when applying ML to clinical applications. For instance, recent work by Roberts et al.²⁷ investigated 2122 studies applying ML to COVID-19 use-cases, finding that none of the models are sufficient for clinical use due to methodological flaws and/or underlying biases. They go on to give many recommendations—some we’ve discussed in the context of MLTRL, and more—which should be reviewed for higher quality medical-ML models and documentation.
Level 6–9 (manufacturing)—Overall these stages proceeded regularly and efficiently for the defect detection product. MLTRL’s embedded switchback from level 9 to 4 proved particularly useful in this lifecycle, both for incorporating feedback from the field and for updating with research progress. On the former, the data distribution shifts from one deployment setting to another significantly affected false-positive versus false-negative calibrations, so this was added as a feature to the CI/CD pipelines. On the latter, the built-in touch points for real-world feedback and data into the continued ML research provided valuable constraints to help guide research, and product managers could readily understand what capabilities could be available for product integration and when (readily communicated with TRL Cards)—for instance, later adding support for video-based inspection for defects, and tooling for end-users to reason about uncertainty estimates (which helps establish trust).
Level 7–9 (medical)—For productization the “neuropathology copilot” was handed off to a partner pharmaceutical company to integrate into their existing software systems. The MLTRL documentation and communication streamlined the technology transfer, which can often be a time-consuming manual process. If not pursuing this path, the product would’ve likely faced many of the medical-ML deployment challenges with model availability and data access; MLTRL cannot overcome the technical challenges of deploying on-premises, but the manifestation of those challenges as performance regressions, data shifts, privacy and ethics concerns, etc. can be mitigated by the system-level checks and strategies MLTRL puts forth.
Advancements in physics engines and graphics processing have advanced AI environment and data-generation capabilities, putting increased emphasis on transitioning models across the simulation-to-reality gap^28,29,30. To develop a computer vision application for automated recycling, we leveraged the Unity Perception³¹ package, a toolkit for generating large-scale datasets for perception-based ML training and validation. We produced synthetic images to complement real-world data sources (Fig. 5). This application exemplifies three important challenges in ML product development that MLTRL helps overcome:
Multiple and disparate data sources are common in deployed ML pipelines yet often ignored in R&D. For instance, upstream data providers can change formats unexpectedly, or a physical event could cause the customer behavior to change. It is nearly impossible to anticipate and design for all potential problems with real-world data and deployment. This computer vision system implemented pipelines and extended test suites to cover open-source benchmark data, real user data, and synthetic data.
Hidden performance degradation can be challenging to detect and debug in ML systems because gradual changes in performance may not be immediately visible. Common reasons for this challenge are that the ML component may be one step in a series. Additionally, local/isolated changes to an ML component’s performance may not directly affect the observed downstream performance. We can see both issues in the illustrated logic diagram for the automated recycling app (Fig. 5). A slight degradation in the initial CV model may not heavily influence the following user input. However, when an uncommon input image appears in the future, the app fails altogether.
Model usage requirements can make or break an ML product. For example, the Netflix “$1M Prize” solution was never fully deployed because of significant engineering costs in real-world scenarios (netflixtechblog.com/netflix-recommendations-beyond-the-5-stars-part-1). For example, engineering teams must communicate memory usage, compute power requirements, hardware availability, network privacy, and latency to the ML teams. ML teams often only understand the statistics or ML theory behind a model but not the system requirements or how it scales.
We next elucidate these challenges and how MLTRL helps overcome them in the context of this project’s lifecycle. This project started at level 4, using largely existing ML methods with a target use case. Specifically, the computer vision (CV) model for object recognition and classification was off-the-shelf, allowing us to bypass levels 0 and 1. Similarly, the synthetic data generation method used Unity Perception, a well-established open-source project. Additionally, the previous project established model training and data pipelines for production, allowing us to bypass level 3.
Level 4—Though previous work allowed us to skip the earlier levels, many challenges arise when combining ML elements that were independently validated and developed. The MLTRL prototype-caliber code checkpoint ensures that the existing code components are validated and helps avoid poorly defined borders and abstractions between components. ML pipelines often grow out of glue code, and our regimented code checkpoints motivate well-architected software that minimizes these danger spots.
Level 5—The problematic “valley of death”, mentioned earlier in the level 5 definitions, is less prevalent in use-cases like this that start at a higher MLTRL level with a specific product deliverable. In this case, the product deliverable was a real-time object recognition and classification of trash for a mobile recycling application. Still, this stage is critical for the requirements and V&V transition. This stage mitigates failure risks due to the disparate data sources integrated at various steps in this CV system and accounted for the end-user compute constraints for mobile computing. Specifically, the TRL cards from earlier stages surfaced potential issues with imbalanced datasets and the need for specific synthetic images. These considerations are essential for the data readiness and testing of V&V in the productization requirements. Data quality and availability issues often present huge blockers because teams discover them too late in the game. Data-readiness is one class of many example issues teams face without MLTRL, as depicted in Fig. 4.
Level 6—We were re-using a well-understood model and deployment pipeline in this use-case, meaning our primary challenge was around data reliability. For the problem of recognizing and classifying trash, building a reliable data source using only real data is almost impossible due to diversity, class imbalance, and annotation challenges. Therefore we chose to develop a synthetic data generator to create training data. At this MLTRL level, we needed to ensure that the synthetic data generator created sufficiently diverse data and exposed the controls needed to alter the data distribution in production. Therefore, we carefully exposed APIs using the Unity Perception package, which allowed us to control lighting, camera parameters, target and non-target object placements and counts, and background textures. Additionally, we ensured that the object labeling matched the real-world annotator instructions and that output data formats matched real-world counterparts. Lastly, we established a set of statistical tests to compare synthetic and real-world data distributions. The MLTRL checks ensured that we understood, and in this case, adequately designed our data sources to meet in-production requirements.
Level 7—From the previous level’s R&D TRL cards and observations, we knew relatively early in productization that we would need to assume bias for the real data sources due to class imbalance and imperfect annotations. Therefore we designed tests to monitor these in the deployed application. MLTRL imposes these critical deployment tests well ahead of deployment, where we can easily overlook ML-specific failure modes.
Level 8—As we suggested earlier, problems that stem from real-world data are near impossible to anticipate and design for, implying the need for level 8 mission-readiness preparations. Given that we were generating synthetic images (with structured domain randomization) to complement the real data, we created tests for different data distribution shifts at multiple points in the classification pipeline. We also implemented thorough shadow tests ahead of deployment to evaluate how susceptible the ML model(s) to performance regressions caused by data. Additionally, we also implemented these as CI/CD tests over various deployment scenarios (or mobile device computing specifications). Without these fully covered, documented, and automated, it would be impossible to pass the Level 8 review and deploy the technology.
Level 9—Post-deployment, the monitoring tests prescribed in Levels 8 and 9, and the three main code quality checkpoints in the MLTRL process help surface hidden performance degradation problems, common with complex pipelines of data flows and various models. The switchbacks depicted in Fig. 4 are typical in CV use-cases. For instance, miscalibrations in models pre-trained on synthetic data and fine-tuned on newer real data can be common yet difficult to catch. However, the level 7 to 4 switchback is designed precisely for these challenges and product improvements.
Computational models and simulation are key to scientific advances at all scales, from particle physics to material design and drug discovery, to weather and climate science, and to cosmology³². Many simulators model the forward evolution of a system (coinciding with the arrow of time), such as the interaction of elementary particles, diffusion of gasses, folding of proteins, or evolution of the universe on the largest scale. The task of inference refers to finding initial conditions or global parameters of such systems that can lead to some observed data representing the final outcome of a simulation. In probabilistic programming³³, this inference task is performed by defining prior distributions over any latent quantities of interest, and obtaining posterior distributions over these latent quantities conditioned on observed outcomes (for example, experimental data) using the Bayes rule. This process, in effect, corresponds to inverting the simulator such that we go from the outcomes toward the inputs that caused the outcomes. In the “Etalumis” project³⁴ (“simulate” spelled backward), we are using probabilistic programming methods to invert existing, large-scale simulators via Bayesian inference. The project is an interdisciplinary collaboration of specialists in probabilistic machine learning, particle physics, and high-performance computing, all essential elements to achieve the project outcomes. Even more, it is a multi-year project spanning multiple countries, companies, university labs, and government research organizations, bringing significant challenges in project management, technology coordination, and validation. Aided by MLTRL, there were several key challenges to overcome in this project that is common in scientific-ML projects:
Integrating with legacy systems is common in scientific and industrial use-cases, where ML methods are applied with existing sensor networks, infrastructure, and codebases. In this case, particle physics domain experts at CERN are using the SHERPA simulator³⁵, a 1 million-line codebase developed over the last two decades. Rewriting the simulator for ML use-cases is infeasible due to the codebase size and buried domain knowledge, and new ML experts would need significant onboarding to gain a working knowledge of the codebase. It is also common to work with legacy data infrastructure, which can be poorly organized for machine learning (let alone preprocessed and clean) and unlikely to have followed best practices such as dataset versioning.
Coupling hardware and software architectures are non-trivial when deploying ML at scale, as performance constraints are often considered in deployment tests well after model and algorithm development, not to mention the expertise is often split across disjoint teams. This can be exacerbated in scientific-ML when scaling to supercomputing infrastructure, and working with massive datasets that can be in the terabytes and petabytes.
Interpretability is often a desired feature yet difficult to deliver and validate in practice. Particularly in scientific ML applications such as this, mechanisms and tooling for domain experts to interpret predictions and models are key for usability (integrating into workflows and building trust).
To this end, we will go through the MLTRL levels one by one, demonstrating how they ensure the above scientific ML challenges are diligently addressed.
Level 0—The theoretical developments leading to Etalumis are immense and well discussed in ref. 34. In particular, the ML theory and methods are in a relatively nascent area of ML and mathematics, probabilistic programming. New territory can present more challenges compared to well-traveled research paths, for instance in computer vision with neural networks. It is thus helpful to have a guiding framework when making a new path in ML research, such as MLTRL where early reviews help theoretical ML projects get legs.
Level 1–2—Running low-level experiments in simple testbeds is generally straightforward when working with probabilistic programming and simulation; in a sense, this easy iteration over experiments is what PPL is designed for. It was additionally helpful in this project to have rich data grounded in physical constraints, allowing us to better isolate model behaviors (rather than data assumptions and noise). The MLTRL requirements documentation is particularly useful for the standard PPL experimentation workflow: model, infer, criticize, repeat (or Box’s loop)³⁶. The evaluation step (i.e. criticizing the model) can be more nuanced than checking summary statistics as in deep learning and similar ML workflows. It is thus a useful practice to write down the criticism methods, metrics, and expected results as verifications for specific research requirements, rather than iterating over Box’s loop without a priori targets. Further, because this research project had a specific target application early in the process (the SHERPA simulator), the project timeline benefited from recognizing simulator-integration constraints upfront as requirements, not to mention data availability concerns, which are often overlooked in early R&D levels. It was additionally useful to have CERN scientists as domain experts in the reviews at these R&D levels.
Level 3—Systems development can be challenging with probabilistic programming, again because it is relatively nascent and much of the out-of-the-box tools and infrastructure are not there as in most ML and deep learning. Here in particular there’s a novel (unproven) approach for systems integration: a probabilistic programming execution protocol was developed to reroute random number draws in the stochastic simulator codebase (SHERPA) to the probabilistic programming system, thus enabling the system to control stochastic choices in SHERPA and run inference on its execution traces, all while keeping the legacy codebase intact! A more invasive method that modifies SHERPA would not have been acceptable. If it were not for MLTRL forcing systems considerations this early in the Etalumis project lifecycle, this could have been an insurmountable hurdle later when multiple codebases and infrastructures come into play. By the same token, systems planning here helped enable the significant HPC scaling later: the team defined the need for HPC support well ahead of actually running HPC, in order to build the prototype code in a way that would readily map to HPC (in addition to local or cloud CPU and GPU). The data engineering challenges in this system’s development nonetheless persist—that is, data pipelines and APIs that can integrate various sources and infrastructures, and normalize data from various databases—although MLTRL helps consider these at an earlier stage that can help inform architecture design.
Level 4—The natural “embedded switchback” from Level 4 to 2 (see the “Methods” section) provided an efficient path toward developing an improved, amortized inference method–i.e., using a computationally expensive deep learning-based inference algorithm to train only once, in order to then do fast, repeated inference in the SHERPA model. Leveraging cyclic R&D methods, the Etalumis project could iteratively improve inference methods without stalling the broader system development, ultimately producing the largest scale posterior inference in a Turing-complete probabilistic programming system. Achieving this scale through iterative R&D along the main project lifecycle was additionally enabled by working with National Energy Research Scientific Computing (NERSC) engineers and their Cori supercomputer to progressively scale smaller R&D tests to the goal supercomputing deployment scenario. Typical ML workflows that follow simple linear progressions^37,38 would not enable ramping up in this fashion, and can actually prevent scaling R&D to production due to a lack of systems engineering processes (like MLTRL) connecting research to deployment.
Level 5—Multi-org international collaborations can be riddled with communication and teamwork issues, in particular at this pivotal stage where teams transition from R&D to application and product development. First, MLTRL as a lingua franca was key to the team effort in bringing Etalumis proof-of-concept into the larger effort of applying it to massive high-energy physics simulators. It was also critical at this stage to clearly communicate end-user requirements across the various teams and organizations, which must be defined in MLTRL requirements docs with V&V measures—the essential science-user requirements were mainly for model and prediction interpretability, uncertainty estimation, and code usability. If there are concerns over these features, MLTRL switchbacks can help to quickly cycle back and improve modeling choices in a transparent, efficient way— generally, in ML projects, these fundamental issues with usability are caught too late, even after deployment. In the probabilistic generative model setting, we’ve defined in Etalumis, Bayesian inference gives results that are interpretable because they include exact locations and processes in the model that are associated with each prediction. Working with ML methods that are inherently interpretable, we are well-positioned to deliver interpretable interfaces for the end-users later in the project lifecycle.
Level 6–9—The standard MLTRL protocol applies in these application-to-deployment stages, with several Etalumis-specific highlights. First, given the significant research contributions in both probabilistic programming and scientific ML, it’s important to share the code publicly. The development and deployment of the open-source code repository PPX (github.com/pyprob/ppx) branched into a separate MLTRL path from the Etalumis path for deployment at CERN. It’s useful to have systems engineering enable a clean separation of requirements, deployments, etc. when there are different development and product lifecycles originating from a common parent project. For example, in this case, it was useful to employ MLTRL switchbacks in the open-sourcing process, isolated from the CERN application paths, in order to add support for additional programming languages so PPX can apply to more scientific simulators—both directions benefited significantly from the data pipelines considerations brought up levels earlier, where open-sourcing required different data APIs and data transformations to enable broad usability. Second, related to the open-source code deliverable and the scientific ML user requirements we noted above, the late stages of MLTRL reviews include higher-level stakeholders and specific end-users, yet again enforcing these scientific usability requirements are met. An example result of this in Etalumis is the ability to output human-readable execution traces of the SHERPA runs and inference, enabling never-before-possible step-by-step interpretability of the black-box simulator.
The scientific ML perspective additionally brings to the forefront an end-to-end data perspective that is pertinent in essentially all ML use-cases: these systems are only useful to the extent they provide comprehensive data analyses that integrate the data consumed and generated in these workflows, from raw domain data to machine-learned models. These data analyses drive reproducibility, explainability, and experiment data understanding, which are critical requirements in scientific endeavors and ML broadly.
Understanding cause and effect relationships are crucial for accurate and actionable decision-making in many settings, from healthcare and epidemiology to economics and government policy development. Unfortunately, standard machine learning algorithms can only find patterns and correlations in data, and as correlation is not causation, their predictions cannot be confidently used for understanding cause and effect. Indeed, relying on correlations extracted from observational data to guide decision-making can lead to embarrassing, costly, and even dangerous mistakes, such as concluding that asthma reduces pneumonia mortality risk³⁹ and that smoking reduces the risk of developing severe COVID-19⁴⁰. Fortunately, there has been much recent development in a field known as a causal inference that can quantitatively make sense of cause and effect from purely observational data⁴¹. The ability of causal inference algorithms to quantify causal impact rests on a number of important checks and assumptions—beyond those employed in standard machine learning or purely statistical methodology—that must be carefully deliberated over during their development and training. These specific checks and assumptions are as follows:
Specifying cause-and-effect relationships between relevant variables: One of the most important assumptions underlying causal inference is the structure of the causal relations between quantities of interest. The gold standard for determining causal relations is to perform a randomized controlled trial, but in most cases, these cannot be employed due to ethical concerns, technological infeasibility, or prohibitive cost. In these situations, domain experts have to be consulted to determine the causal relationships. It is important in these situations to carefully address the manner in which such domain knowledge was extracted from experts, the number and diversity of experts involved, the amount of consensus between experts, and so on. The need for careful documentation of this knowledge and its periodic review is made clear in the MLTRL framework, as we shall see below.
Identifiability: Another vital component of building causal models is whether the causal question of interest is identifiable from the causal structure specified for the model together with observational (and sometimes experimental) data.
Adjusting for and monitoring confounding bias: An important aspect of causal model performance, not present in standard machine learning algorithms, is confounding bias adjustment. The standard approach is to employ propensity score matching to remove such bias. However, the quality of bias adjustment achieved in any specific instance with such propensity-based matching methods needs to be checked and documented, with alternate bias adjusting procedures required if appropriate levels of bias adjustment are not achieved⁴².
Sensitivity analysis: As causal estimates are based on generally untestable assumptions, such as observing all relevant confounders, it is vital to determine how sensitive the resulting predictions are to potential violations of these assumptions.
Consistency: It is crucial to understand if the learned causal estimate provably converges to the true causal effect within the limit of an infinite sample size. However, causal models cannot be validated by standard held-out tests, but rather require randomization or special data collection strategies to evaluate their predictions^43,44.
The MLTRL framework makes transparent the need to carefully document and defends these assumptions, thus ensuring the safe and robust creation, deployment, and maintenance of causal models. We elucidate this with recent work by Richens et al.⁴⁵, developing a causal approach to computer-assisted diagnosis which outperforms previous purely machine learning-based methods. To this end, we will go through the MLTRL levels one by one, demonstrating how they ensure the above specific checks and assumptions are naturally accounted for. This should provide a blueprint for how to employ the MLTRL levels in other causal inference applications.
Level 0—When initially faced with a causal inference task, the first step is always to understand the causal relationships between relevant variables. For instance, in ref. 45, the first step toward building the diagnostic model was specifying the causal relationships between the diverse set of risk factors, diseases, and symptoms included in the model. To learn these relations, doctors and healthcare professionals were consulted to employ their expansive medical domain knowledge which was robustly evaluated by additional independent groups of healthcare professionals. The MLTRL framework ensured this issue is dealt with and documented correctly, as such knowledge is required to progress from Level 0; failure to do this has plagued similar healthcare AI projects⁴⁶. The next step of any causal analysis is to understand whether the causal question of interest is uniquely identifiable from the causal structure specified for the model together with observational and experimental data. In this medical diagnosis example, identification was crucial to establish, as the causal question of interest, “would the observed symptoms not be present had a specific disease been cured?”, was highly non-trivial. Again, MLTRL ensures this vital aspect of model building is carefully considered, as a mathematical proof of identifiability would be required to graduate from Level 0. With both the causal structure and identifiability result in hand, one can progress to Level 1.
Level 1—At this level, the goal is to take the estimand for the identified causal question of interest and devise a way to estimate it from data. To do this one will need efficient ways to adjust for confounding bias. The standard approach is to employ propensity score-based methods to remove such bias when the target decision is binary, and use multi-stage ML models adhering to the assumed causal structure⁴⁷ for continuous target decisions (and high-dimensional data in general). However, the quality of bias adjustment achieved in any specific instance with propensity-based matching methods needs to be checked and documented, with alternate bias adjusting procedures required if appropriate levels of bias adjustment are not achieved⁴². As above, MLTRL ensures transparency and adherence to this important aspect of causal model development, as without it a project cannot graduate from Level 1. Even more, MLTRL ensures tests for confounding bias are developed early on and maintained throughout later stages of deployment. Still, in many cases, it is not possible to completely remove confounding in the observed data. TRL Cards offer a transparent way to declare specific limitations of a causal ML method.
Level 2—PoC-level tests for causal models must go beyond that of typical ML models. As discussed above, to ensure the estimated causal effects are robust to the assumptions required for their derivation, sensitivity to these assumptions must be analyzed. Such sensitivity analysis is often limited to R&D experiments or a post-hoc feature of ML products. MLTRL on the other hand requires this throughout the lifecycle as components of ML test suites and gated reviews. In the case of causal ML, best practice is to employ sensitivity analysis for this robustness check⁴⁸. MLTRL ensures this check is highlighted and adhered to, and no model will end up graduating Level 2–let alone being deployed—unless it is passed.
Level 3—Coding best practices, as in general ML applications.
Level 4–5—There are additional tests to consider when taking causal models from research to production, in particular at Level 4—proof of concept demonstration in a real scenario. Consistency, for example, is an important property of causal methods that informs us whether the method provably converges to the true causal graph within the limit of infinite sample size. Quantifying consistency in the test suite is critical when datasets change from controlled laboratory settings to open-world, and when the application scales. And PoC validation steps are more efficient with MLTRL because the process facilitates early specification of the evaluation metric for a causal model in Level 2. Causal models cannot be validated by standard held-out tests, but rather require randomization or special data collection strategies to evaluate their predictions^43,44. Any difficulty in evaluating the model’s predictions will be caught early and remedied.
Level 6–9—With the causal ML components of this technology developed reliably in the previous levels, the rest of the levels developing this technology focused on general medical-ML deployment challenges. For the most part, data governance, privacy, and management that was detailed earlier in the neuropathology MLTRL use-case, as well as the on-premises deployment.
The Cameras for Allsky Meteor Surveillance (CAMS) project⁴⁹, established in 2010 by NASA, uses hundreds of off-the-shelf CCTV cameras to capture meteor activity in the night sky. Initially, resident scientists would retrieve hard disks containing video data captured each night and perform manual triangulation of tracks or streaks of light in the night sky, and compute a meteor’s trajectory, orbit, and light curve. Each solution was manually classified as a meteor or not (i.e., planes, birds, clouds, etc.). In 2017, a project run by the Frontier Development Lab⁵⁰ (The NASA Frontier Development Lab and partners open-source the code and data via the SpaceML platform: spaceml.org), the AI accelerator for NASA and ESA, aimed to automate the data processing pipeline and replicate the scientists thought process to build an ML model that identifies meteors in the CAMS project^51,52. The data automation led to orders of magnitude improvements in the operational efficiency of the system and allowed new contributors and amateur astronomers to start contributing to meteor sightings. Additionally, a novel web tool allowed anybody anywhere to view the meteors detected the previous night. The CAMS camera system has had a six-fold global expansion of the data capture network, discovered ten new meteor showers, contributed towards instrumental evidence of previously predicted comets, and helped calculate parent bodies of various meteor showers. CAMS utilized the MLTRL framework to progress as described:
Level 1—Understanding the domain and data is a prerequisite for any ML development. Extensive data exploration elucidated visual differences between objects in the night sky such as meteors, satellites, clouds, tail lights of planes, and light from the eyes of cats peering into cameras, trees, and other tall objects visible in the moonlight. This step helped (1) understand the visual properties of meteors that later defined the ML model architecture, and (2) mitigate the impact of data imbalance by proactively developing domain-oriented strategies. The results are well-documented on a datasheet associated with the TRL card and discussed at the stage review. This MLTRL documentation forced us to consider data sharing and other privacy concerns at this early conceptualization stage, which is certainly relevant considering CAMS is for open-source and gathering data from myriad sources.
Level 2–3—The agile and non-monotonic (or non-linear) development prescribed by MLTRL allowed the team to first develop an approximate end-to-end pipeline that offered a path to ML model deployment and quick turnaround time to incorporate feedback from the regular gated reviews. Then, with relatively quicker experimentation, the team could improve on the quality of not just the ML model, but also scale up the systems development simultaneously in a non-monotonic development cycle.
Level 4—With the initial pipeline in place, scalable training of baselines and initial models on real challenging datasets ensued. Throughout the levels, the MLTRL gated reviews were essential for making efficient progress while ensuring robustness and functionality that meets stakeholder needs. At this stage we highlight specific advantages of the MLTRL review processes that had an instrumental effect on the project’s success: With the required panel of mixed ML researchers and engineers, domain scientists, and product managers, the stage 4 reviews stressed the significance of numerical improvements and comparison to existing baselines and helped identify and overcome issues with data imbalance. The team likely would have overlooked these approaches without the review from peers in diverse roles and teams. In general, the evolving panel of reviewers at different stages of the project was essential for covering a variety of verification and validation measures—from helping mitigate data challenges to open-source code quality.
Level 5—To complete this R&D-to-productization level, a novel web tool called the NASA CAMS Meteor Shower Portal (meteorshowers.seti.org) was created that allowed users to view meteor shower activity from the previous night and verify meteor predictions generated by the ML model. This app development was valuable for A/B testing, validating detected meteors and classifying new meteor showers with human–AI interaction, and demonstrating real-world utility to stakeholders in review. ML processes without MLTRL miss out on this valuable development by overlooking the need for such a demo tool.
Level 6—Application development was naturally driven by end-user feedback from the web app in level 5—without MLTRL it’s unlikely the team would be able to work with early productization feedback. With almost real-time feedback coming in daily, newer methods for improving the robustness of meteor identification led to researching and developing a unique augmentation technique, resulting in the state-of-the-art performance of the ML model. Further application development led to incorporating features that were in demand by users of the NASA CAMS Meteor Shower Portal: including celestial reference points through constellations, adding the ability to zoom in/out and (un)cluster showers, and providing tooling for scientific communication. The coordination of these features into a product-caliber codebase resulted in the release of the NASA CAMS Meteor Shower Portal 2.0 which was built by a team of citizen scientists—again we found the specific checkpoints in the MLTRL review were crucial for achieving these goals.
Level 7—Integration was particularly challenging in two ways. First, integrating the ML and data engineering deliverables with the existing infrastructure and tools of the larger CAMS system, which had started development years earlier with other teams in partner organizations, required quantifiable progress for verifying the tech-readiness of ML models and modules. The use of technology readiness levels provided a clear and consistent metric for the maturity of the ML and data technologies, making for clear communication and efficient project integration. Without MLTRL it is difficult to have a conversation, let alone make progress, towards integrating ML/AI and data subsystems and components. Second, integrating open-source contributions into the main ML subsystem was a significant challenge alleviated with diligent verification and validation measures from MLTRL, as well as quantifying robustness with ML testing suites (using scoring measures like that of the ML Testing Rubric²⁰, and devising a checklist based on metamorphic testing¹⁸).
Level 8—CAMS, like many datasets in practice, consists of a smaller labeled subset and a much larger unlabeled set. In an attempt to additionally increase the robustness of the ML subsystem ahead of “mission-readiness”, we looked to active learning^53,54 techniques to leverage the unlabeled data. Models using an initial version of this approach, where results of the active learning provided “weak” labels, resulted in consumption of the entire decade-long unlabeled data collected by CAMS and slightly higher scores on deployment tests. Active learning showed to be a promising feature and was switched back to level 7 for further development towards the next deployment version, so as not to delay the rest of the project.
Level 9—The ML components in CAMS require continual monitoring for model and data drifts, such as changes in weather, smoke, and cloud patterns that affect the view of the night sky. The data drifts may also be specific to locations, such as fireflies and bugs in CAMS Australia and New Zealand stations which appear as false positives. The ML pipeline is largely automated with CI/CD, runs regular regression tests, and production of benchmarks. Manual intervention can be triggered when needed, such as sending low-confidence meteors for verification to scientists in the CAMS project. The team also regularly releases the code, models, and web tools on the open-source space sciences and exploration ML toolbox, SpaceML (spaceml.org). Through the SpaceML community and partner organizations, CAMS continually improves with feature requests, debugging, and improving data practices, while tracking progress with standard software release cycles and MLTRL documentation.
Software engineering (SWE) practices vary significantly across domains and industries. Some domains, such as medical applications, aerospace, or autonomous vehicles rely on a highly rigorous development process which is required by regulations. Other domains, for example, advertising and e-commerce are not regulated and can employ a lenient approach to development. ML development should at minimum inherit the acceptable software engineering practices of the domain. There are, however, several key areas where ML development stands out from SWE, adding its own unique challenges which even the most rigorous SWE practices are not able to overcome.
For instance, the behavior of ML systems is learned from data, not specified directly in code. The data requirements around ML (i.e., data discovery, management, and monitoring) add significant complexity not seen in other types of SWE. There are many benefits to using a data-oriented architecture (DOA)⁴⁶ with the data-first workflows and management practices prescribed in MLTRL. DOA aims to make the data flowing between elements of business logic more explicit and accessible with a streaming-based architecture rather than the micro-service architectures that are standard in software systems. One specific benefit of DOA is making data available and traceable by design, which helps significantly in the ML logging challenges and data governance needs we discussed in Levels 7–9. Moreover, MLTRL highlights data-related requirements along every step to ensure that the development process considers data readiness and availability.
There are an array of ML-specific failure modes that must be carefully addressed before ML algorithms are deployed. For example, models become miscalibrated due to subtle data distributional shifts in the deployment setting, resulting in models that are more confident in predictions than they should be. MLTRL helps define ML-specific testing considerations (levels 5 and 7) to help surface ML-specific failure modes early. ML opens up new threat vectors across the whole deployment workflow that otherwise are not risks in software systems: for example, a poisoning attack to contaminate the training phase of ML systems, or membership inference to see if a given data record was part of the model’s training. MLTRL considers these threat vectors and suggests relevant risk identification during the prototyping and productization phases. More generally, ML codebases have all the problems for regular code, plus ML-specific issues at the system level, mainly as a consequence of added complexity and dynamism. The resulting entanglement, for instance, implies that the SWE practice of making isolated changes is often not feasible—Scully et al.⁵⁵ refer to this as the “changing anything changes everything” principle. Given this consideration, typical SWE change-management is insufficient. Furthermore, ML systems almost necessarily increase the technical debt; package-level refactoring is generally sufficient for removing technical debt in software systems, but this is not the case in ML systems.
These factors and others suggest that inherited software engineering and management practices of a given domain are insufficient for the successful development of robust and reliable ML systems. But it is not trading off one for the other: MLTRL can be used in synergy with the existing, industry-standard software engineering practices such as agile⁵⁶ and waterfall⁵⁷ to handle unique challenges of ML development. Because ML applications are a category of software, all best practices of building and operating software should be extended when possible to the ML application. Practices like version control, comprehensive testing, continuous integration, and continuous deployment are all applicable to ML development. MLTRL provides a framework that helps extend SWE building and operating practices that are acceptable in a given domain to tackle the unique challenges of ML development.
A recent case study from Microsoft Research³⁸ similarly identifies a few themes describing how ML is not equal to software engineering, and recommends a linear ML workflow with steps for data preparation through modeling and deploying. They define an effective workflow for isolated development of an ML model, but this approach does not ensure the technology is actually improving in quality and robustness. Their process should be repeated at progressive stages of development in the broader ML and data technology lifecycle. If applied in the MLTRL framework, the specific ingredients of the ML model workflow—that is, people, software, tests, objectives, etc. —evolve over time and subsequent stages as the technologies mature.
There exist many recommended workflows for specific ML methods and areas of pipelines. For instance, a more iterative process for Bayesian ML⁵⁸ and even more specifically for probabilistic programming³⁶, a data mining process defined in 2000 that remains widely used⁵⁹, others for describing data iterations⁶⁰, and human–computer interaction cycles⁶¹. In these recommended workflows and others, there is an important distinction between their cycles and “switchback” mechanisms in MLTRL. Their cycles suggest generically iterating over a data-modeling–evaluation–deployment process. Switchbacks, on the other hand, are specific, purpose-driven workflows for dialing part(s) of a project to an earlier stage—this does not simply mean going back and training the model on more data, but rather switching back regresses the technology’s maturity level (e.g. from level 5 to level 3) such that it must again fulfill the level-by-level requirements, evaluations, and reviews. See the “Methods” section for more details on MLTRL switchbacks. In general, iteration is an important part of data, ML, and software processes. MLTRL is unique from the other recommended processes in many ways, and perhaps most importantly because it considers data flows and ML models in the context of larger systems. These isolated processes (that are specific to e.g. modeling in prototype development or data wrangling in application development) are synergistic with MLTRL because they can be used within each level of the larger lifecycle or framework. For example, the Bayesian modeling processes^36,58 we mentioned above are really useful to guide developers of probabilistic ML approaches. But there are important distinctions between executing these modeling steps and cycles in a well-defined prototyping environment with curated data and minimal responsibilities, versus a production environment riddled with sparse and noisy data, that interacts with the physical world in non-obvious ways, and can carry expensive (even hidden) consequences. MLTRL provides the necessary, holistic context and structure to use these and other development processes reliably and responsibly.
Also related to our work, Google teams have proposed ML testing recommendations²⁰ and validated the data fed into ML systems⁶². For NLP applications, typical ML testing practices struggle to translate to real-world settings, often overestimating performance capabilities. An effective way to address this is devising a checklist of linguistic capabilities and test types, as in ref. 17–interestingly their test suite was inspired by metamorphic testing, which we suggested earlier in Level 7 for testing systems AI integrations. A survey by Paleyes et al.⁴⁶ goes over numerous case studies to discuss challenges in ML deployment. They similarly pay special attention to the need for ethical considerations, end-user trust, and extra security in ML deployments. On the latter point, Kumar et al.⁶³ provide a table thoroughly breaking down new threat vectors across the whole ML deployment workflow (some of which we mentioned above). These works, notably the ML security measures and the quantification of an ML test suite in a principled way—i.e., that does not use misguided heuristics such as code coverage—are valuable to include in any ML workflow including MLTRL, and are synergistic with the framework we’ve described in this paper. These analyses provide useful insights, but they do not provide a holistic, regimented process for the full ML lifecycle from R&D through deployment. An end-to-end approach is suggested by Raji et al.⁶⁴, but only for the specific task of auditing algorithms; components of AI auditing are mentioned in Level 7, and covered throughout the review processes.
Sculley et al.⁵⁵ go into more ML debt topics such as undeclared consumers and data dependencies and go on to recommend an ML Testing Rubric as a production checklist²⁰. For example, testing models by a canary process before serving them in production. This, along with similar shadow testing we mentioned earlier, is common in autonomous ML systems, notably robotics and autonomous vehicles. They explicitly call out tests in four main areas (ML infrastructure, model development, features and data, and monitoring of running ML systems), some of which we discussed earlier. For example, tests that the training and serving features compute the same values; a model may train on logged processes or user input, but is then served on a live feed with different inputs. In addition to the Google ML Testing Rubric, we advocate metamorphic testing: a SWE methodology for testing a specific set of relations between the outputs of multiple inputs. True to the checklists in the Google ML Testing Rubric and in MLTRL, metamorphic testing for ML can have a codified list of metamorphic relations¹⁸.
In domains such as healthcare, there has been the introduction of similar checklists for data readiness—for example, to ensure regulatory-grade real-world-evidence (RWE) data quality⁶⁵— yet these are nascent and not yet widely accepted. Applying AI in healthcare has led to developing guidance for regulatory protocol, which is still a work in progress. Larson et al.⁶⁶ provide a comprehensive analysis for medical imaging and AI, arriving at several regulatory framework recommendations that mirror what we outline as important measures in MLTRL: e.g., detailed task elements such as pitfalls and limitations (surfaced on TRL Cards), clear definition of an algorithm relative to the downstream task, defining the algorithm “capability” (Level 5), real-world monitoring, and more.
D’amour et al.¹⁹ dive into the problem we noted earlier about model miscalibration. They point to the trend in machine learning to develop models relatively isolated from the downstream use and larger system, resulting in underspecification that handicaps practical ML pipelines. This is largely problematic in deep learning pipelines, but we have also noted this risk in the case of causal inference applications. Suggested remedies include stress tests—empirical evaluations that probe the model’s inductive biases on practically relevant dimensions—and in general the methods we define in Level 7.
MLTRL has been developed, deployed, iterated, and validated in myriad environments, as demonstrated by the previous examples and many others. Nonetheless, we strongly suggest that MLTRL not be viewed as a cure-all for machine learning systems engineering. Rather, MLTRL provides mechanisms to better enable ML practitioners, teams, and stakeholders to be diligent and responsible with these technologies and data. That is, one cannot implement MLTRL in an organization and turn a blind eye to the many data, ML, and integration challenges we’ve discussed here. MLTRL is analogous to a pilot’s checklist, not autopilot.
MLTRL is intended to be complementary to existing software development methodologies, not replace or alter them. Specifically, whether the team uses agile or waterfall methods, MLTRL can be adopted to help define and structure phases of the project, as well as the success criteria of each stage. In the context of the software development process, the purpose of MLTRL is to help the team minimize the technical dept and risk associated with the delivery of an ML application by helping the development team ask necessary questions.
We discussed many data challenges and approaches in the context of MLTRL and should highlight again the importance of data considerations in any ML initiative. The data availability and quality can severely limit the ability to develop and deploy ML, whether MLTRL is used or not. It is again the responsibility of the ML practitioners, teams, and stakeholders to gather, use, and distribute data in safe, legal, and ethical ways. MLTRL helps do so with rigor and transparency, but again is not a solution for data bias. We recommend these recent works on data bias in ML^{67,68,69,70,71}. Further, ML/AI ethics is a continuously evolving, multidisciplinary space—see ref. 5. MLTRL aims to prioritize ethics considerations at each level of the framework, and would do well to also evolve over time with the broader ML/AI ethics developments.
We have described Machine Learning Technology Readiness Levels (MLTRL), an industry-hardened systems engineering framework for robust, reliable, and responsible machine learning. MLTRL is derived from the processes and testing standards of spacecraft development, yet lean and efficient for ML, data, and software workflows. Examples from several organizations across industries demonstrate the efficacy of MLTRL for AI and ML technologies, from research and development through productization and deployment, in important domains such as healthcare and physics, with emphasis on data readiness amongst other critical challenges. Our aim is for MLTRL works in synergy with recent approaches in the community focused on diligent data readiness, privacy and security, and ethics. Even more, MLTRL establishes a much-needed lingua franca for the AI ecosystem, and broadly for AI in the worlds of science, engineering, and business. Our hope is that our systems framework is adopted broadly in AI and ML organizations, and that “technology readiness levels” become common nomenclature across AI stakeholders— from researchers and engineers to sales-people and executive decision-makers.
At the end of each stage is a dedicated review period: (1) present the technical developments along with the requirements and their corresponding verification measures and validation steps, (2) make key decisions on path(s) forward (or backward) and timing, and (3) debrief the process (Debriefing (or: retrospectives, formal inquiry, final report, etc.) is a common process, used to improve future performance of projects in project management. See the following reference for more details⁷². MLTRL should include regular debriefs and meta-evaluations such that process improvements can be made in a data-driven, efficient way (rather than an annual meta-review). MLTRL is a high-level framework that each organization should operationalize in a way that suits their specific capabilities and resources.). As in the gated reviews defined by TRL used by NASA, DARPA, etc., MLTRL stipulates specific criteria for review at each level, as well as calling out specific key decision points (noted in the level descriptions above). The designated reviewers will “graduate” the technology to the next level, or provide a list of specific tasks that are still needed (ideally with quantitative remarks). After graduation at each level, the working group does a brief post-mortem; we find that a quick day or two pays dividends in cutting away technical debt and improving team processes. Regular gated reviews are essential for making efficient progress while ensuring robustness and functionality that meets stakeholder needs. There are several important mechanisms in MLTRL reviews that are specifically useful with AI and ML technologies: First, the review panels evolve over a project lifecycle, as noted below. Second, MLTRL prescribes that each review runs through an AI ethics checklist defined by the organization; it is important to repeat this at each review, as the review panel and stakeholders evolve considerably over a project lifecycle. As previously described in the levels definitions, including ethics reviews as an integral part of early system development is essential for informing model specifications and avoiding unintended biases or harm⁷³ after deployment.
In Fig. 2 we succinctly showcase a key deliverable: TRL Cards. The model cards proposed by Google⁷⁴ are a useful development for external user-readiness with ML. On the other hand, our TRL Cards are more information-dense, like datasheets for medical devices and engineering tools. These serve as “report cards” that grow and improve upon graduating levels and provide a means of inter-team and cross-functional communication. The content of a TRL Card is roughly in two categories: project info, and implicit knowledge. The former clearly states info such as project owners and reviewers, development status, and semantic versioning—not just for code, but also for models and data. In the latter category are specific insights that are typically siloed in the ML development team but should be communicated to other stakeholders: modeling assumptions, dataset biases, corner cases, etc. With the spread of AI and ML in critical application areas, we are seeing domain expert consortiums defining AI reporting guidelines—e.g., Rivera et al.⁷⁵ calling for clinical trial reports for interventions involving AI – which will greatly benefit from the use of our TRL reporting cards. We stress that these TRL Cards are key for the progression of projects, rather than documentation afterthoughts. The TRL Cards thus promote transparency and trust, within teams and across organizations. TRL Card templates will be open-sourced upon publication of this work, including methods for coordinating use with other reporting tools such as “Datasheets for Datasets”⁷⁶.
Identifying and addressing risks in a software project is not a new practice. However, akin to the MLTRL roots in spacecraft engineering, the risk is a “first-class citizen” here. In the definition of technical and product requirements, each entry has a calculation of the form risk = p(failure) × value, where the value of a component is an integer 1−10. Being diligent about quantifying risks across the technical requirements is a useful mechanism for flagging ML-related vulnerabilities that can sometimes be hidden by layers of other software. MLTRL also specifies that risk quantification and testing strategies are required for sim-to-real development. That is, there is nearly always a non-trivial gap in transferring a model or algorithm from a simulation testbed to the real world. Requiring explicit sim-to-real testing steps in the workflow helps mitigate unforeseen (and often hazardous) failures. Additionally, the comprehensive ML test coverage that we mention throughout this paper is a critical strategy for mitigating risks and uncertainties: ML-based system behavior is not easily specified in advance, but rather depends on the dynamic qualities of the data and on various model configuration choices²⁰.
We observe many projects benefit from cyclic paths, dialing components of technology back to a lower level. Our framework not only encourages cycles, but we also make them explicit with “switchback mechanisms” to regress the maturity of specific components in an ML/AI system:
Discovery switchbacks occur as a natural mechanism—new technical gaps are discovered through systems integration, sparking later rounds of component development⁷⁷. These are most common in the R&D phase for switching back one or two levels, and also a larger leap back across the “product handoff” gap from level 6 to 3—for example, a computer vision algorithm is only performant on a certain class of camera that is not necessarily available in production, so the algorithm must switchback to validate proof-of-concept on the lower grade camera.
Review switchbacks result from gated reviews, where specific components or larger subsystems may be dialed back to earlier levels. This switchback is one of the “key decision points” in the MLTRL project lifecycle (as noted in the Levels definitions) and is often a decision driven by business needs and timing rather than technical concerns (for instance when mission priorities and funds shift). This mechanism is common from levels 6/7 to 4, which stresses the importance of this R&D to the product transition phase (see Fig. 4 (left)).
Embedded switchbacks are predefined in the MLTRL process. Namely one switchback to move a proof-of-concept technology (at Level 4) back to proof-of-principle development (Level 2), and another for switching back from deployment (9) to proof-of-concept (4). In complex systems, particularly with ML and data-driven technologies, these built-in loops help mitigate technical debt and overcome other inefficiencies such as noncomprehensive V&V steps.
The three classes of switchbacks are described in Figs. 4, 3 and throughout the various application results. Without these built-in mechanisms for cyclic development paths, it can be difficult and inefficient to build systems of modules and components at varying degrees of maturity. Contrary to the traditional thought that switchback events should be suppressed and minimized, in fact, they represent a natural and necessary part of the complex technology development process—efforts to eliminate them may stifle important innovations without necessarily improving efficiency. This is a fault of the standard monotonic approaches in ML/AI projects, stage-gate processes, and even the traditional TRL framework.
It is also important to note that most projects do not start at Level 0; very few ML companies engage in this low-level theoretical research. For example, a team looking to use an off-the-shelf object recognition model could start that technology at Level 3, and proceed with thorough V&V for their specific datasets and use-cases. However, no technology can skip levels after the MLTRL process has been initiated. The industry default (that is, without implementing MLTRL) is to ignorantly take pretrained models, run fine tuning on their specific data, and jump to deployment, effectively skipping Levels 5–7. These patterns are shown in Fig. 4. Additionally, we find it advantageous to incorporate components from other high-TRL ranking projects while starting new projects; MLTRL makes the verification and validation (V&V) step straightforward for integrating previously developed ML components.
As suggested earlier, much of the practical value of MLTRL comes at the transition between levels. More precisely, MLTRL manages these oft-neglected transitions explicitly as evolving teams, objectives, and deliverables. For instance, the team (or working group) at Level 3 is mostly AI Research Engineers, but at Level 6 is mixed Applied AI/SW Engineers mixed with product managers and designers. Similarly, the review panels evolve from level to level, to match the changing technology development objectives. What the reviewers reference similarly evolves: notice in the level definitions that technical requirements and V&V guide early stages, but at and after Level 6 the product requirements and V&V takeover—naturally, the risk quantification and mitigation strategies evolve in parallel. Regarding the deliverables, notably TRL Cards and risk matrices⁷⁸ (to rank and prioritize various science, technical, and project risks), the information develops and evolves over time as the technology matures.
By defining technology maturity in a quantitative way, MLTRL enables teams to accurately and consistently define their ML progress metrics. Notably industry-standard “objectives and key results” (OKRs) and “key performance indicators” (KPIs)⁷⁹ can be defined as achieving certain readiness levels in a given period of time; this is a preferable metric in essentially all ML systems which consist of much more than a single performance score to measure progress. Even more, a meta-review of MLTRL progress over multiple projects can provide useful insights at the organization level. For example, analysis of the time-per-level and the most frequent development paths/cycles can bring to light operational bottlenecks. Compared to conventional software engineering metrics based on sprint stories and tickets, or time-tracking tools, MLTRL provides a more accurate analysis of ML workflows.
A distinct advantage of MLTRL in practice is the nomenclature: an agreed-upon grading scheme for the maturity of an AI technology, and a framework for how/when that technology fits within a product or system, enabling everyone to communicate effectively and transparently. MLTRL also acts as a gate for interpretability and explainability–at the granularity of individual models and algorithms, and more crucially from a holistic, systems standpoint. Notably, the DARPA XAI (DARPA Explainable Artificial Intelligence (XAI)) program advocates for this advance in developing AI technologies; they suggest interpretability and explainability are necessary at various locations in an AI system to be sufficient for deployment as an AI product, otherwise leading to issues with ethics and bias.
How to design a reliable system from unreliable components has been a guiding question in the fields of computing and intelligence⁸⁰. In the case of ML/AI systems, we aim to build reliable systems with myriad unreliable components: noisy and faulty sensors, human and AI error, and so on. There is thus significant value to quantifying the myriad uncertainties, propagating them throughout a system, and arriving at a notion or measure of reliability. For this reason, although MLTRL applies generally to ML/AI methods and systems, we advocate for methods in the class of probabilistic ML, which naturally represent and manipulate uncertainty about models and predictions²⁵. These are Bayesian methods that use probabilities to represent aleatoric uncertainty, measuring the noise inherent in the observations, and epistemic uncertainty, accounting for uncertainty in the model itself (i.e., capturing our ignorance about which model generated the data). In the simplest case, an uncertainty-aware ML pipeline should quantify uncertainty at the points of sensor inputs or perception, prediction or model output, and decision or end-user action—McAllister et al.²⁶ suggest this with Bayesian deep learning models for safer autonomous vehicle pipelines. We can achieve this sufficiently well in practice for simple systems. However, we do not yet have a principled, theoretically grounded, and generalizable way of propagating errors and uncertainties downstream and throughout more complex AI systems—i.e., how to integrate different software, hardware, data, and human components while considering how errors and uncertainties propagate through the system. This is an important direction for our future work.
Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study. For the presented examples reflecting other studies, datasets may be found in the corresponding links or references provided in the text (if applicable). Please contact the corresponding author(s) with questions or concerns.
Code sharing is not applicable to this article as no code was produced. Otherwise, implementation materials for this work are included in an open-source repository at https://github.com/ai-infrastructure-alliance/mltrl, along with a full MLTRL home at ai-infrastructure.org/mltrl).
Henderson, P. et al. Deep reinforcement learning that matters. In Proc. AAAI Conference on Artificial Intelligence (2018).
de la Tour, A., Portincaso, M., Blank, K. & Goeldel, N. The Dawn of the Deep Tech Ecosystem. Technical Report (The Boston Consulting Group, 2019).
NASA. The NASA Systems Engineering Handbook (NASA, 2003).
United States Department of Defense. Defense Acquisition Guidebook (U.S. Department of Defense, 2004).
Leslie, D. Understanding artificial intelligence ethics and safety: A guide for the responsible design and implementation of AI systems in the public sector. The Alan Turing Institute. https://doi.org/10.5281/zenodo.3240529 (2019).
Lavin, A. & Renard, G. Technology readiness levels for AI & ML. In ICML Workshop on Challenges Deploying ML Systems (2020).
Dasu, T. & Johnson, T. Exploratory Data Mining and Data Cleaning (John Wiley & Sons, 2003).
Janssen, M., Brous, P., Estevez, E., Barbosa, L. & Janowski, T. Data governance: organizing data for trustworthy artificial intelligence. Gov. Inf. Q. 37, 101493 (2020).
Article  Google Scholar 
Shahriari, B., Swersky, K., Wang, Z., Adams, R. P. & De Freitas, N. Taking the human out of the loop: a review of bayesian optimization. Proc. IEEE 104, 148–175 (2015).
Article  Google Scholar 
Boehm, B. W. Verifying and validating software requirements and design specifications. IEEE Softw. 1, 75 (1984).
Article  Google Scholar 
Ramakrishnan, G., Nori, A., Murfet, H. & Cameron, P. Towards compliant data management systems for healthcare ML. Preprint at ArXiv: abs/2011.07555 (2020).
Bhatt, U. et al. Explainable machine learning in deployment. In Proc. 2020 Conference on Fairness, Accountability, and Transparency (2020).
Li, T., Sahu, A. K., Talwalkar, A. & Smith, V. Federated learning: challenges, methods, and future directions. IEEE Signal Process. Mag. 37, 50–60 (2020).
Google Scholar 
Ryffel, T. et al. A generic framework for privacy preserving deep learning. In NeurIPS Workshop (PPML, 2018).
Madry, A., Makelov, A., Schmidt, L., Tsipras, D. & Vladu, A. Towards deep learning models resistant to adversarial attacks. In The Sixth International Conference on Learning Representations (ICLR, 2018).
Zhao, Z., Dua, D. & Singh, S. Generating natural adversarial examples. In International Conference on Learning Representations (2018).
Ribeiro, M. T., Wu, T., Guestrin, C. & Singh, S. Beyond accuracy: behavioral testing of NLP models with CheckList. In Proc. ACL (2020).
Xie, X. et al. Testing and validating machine learning classifiers by metamorphic testing. J. Syst. Softw. 844, 544–558 (2011).
Article  Google Scholar 
D’Amour, A. et al. Underspecification presents challenges for credibility in modern machine learning. Preprint at ArXiv: abs/2011.03395 (2020).
Breck, E., Cai, S., Nielsen, E., Salib, M. & Sculley, D. The ML Test Score: a rubric for ML production readiness and technical debt reduction. In 2017 IEEE International Conference on Big Data (Big Data) 1123–1132 (2017).
Botchkarev, A. A new typology design of performance metrics to measure errors in machine learning regression algorithms. Interdiscip. J. Inf. Knowl. Manag. 14, 045–076 (2019).
Google Scholar 
Naud, L. & Lavin, A. Manifolds for unsupervised visual anomaly detection. Preprint at ArXiv: abs/2006.11364 (2020).
Schulam, P. & Saria, S. Reliable decision support using counterfactual models. In NeurIPS 2017 (2017).
Towards trustable machine learning. Nat. Biomed. Eng. 2, 709–710 (2018).
Ghahramani, Z. Probabilistic machine learning and artificial intelligence. Nature 521, 452–459 (2015).
Article  ADS  CAS  Google Scholar 
McAllister, R. et al. Concrete problems for autonomous vehicle safety: advantages of bayesian deep learning. In IJCAI (2017).
Roberts, M. et al. Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans. Nat. Mach. Intell. 3, 199–217 (2021).
Article  Google Scholar 
Tobin, J. et al. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 23–30 (2017).
Juliani, A. et al. Unity: a general platform for intelligent agents. Preprint at ArXiv: abs/1809.02627 (2018).
Hinterstoisser, S., Pauly, O., Heibel, H., Marek, M. & Bokeloh, M. An annotation saved is an annotation earned: using fully synthetic training for object detection. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) 2787–2796 (IEEE Computer Society, 2019).
Borkman, S. et al. Unity perception: generate synthetic data for computer vision. Preprint at ArXiv: CoRR abs/2107.04259 (2021).
Cranmer, K., Brehmer, J. & Louppe, G. The frontier of simulation-based inference. Proc. Natl Acad. Sci. USA 117, 30055–30062 (2020).
Article  ADS  MathSciNet  CAS  Google Scholar 
van de Meent, J.-W., Paige, B., Yang, H. & Wood, F. An introduction to probabilistic programming. Preprint at ArXiv: abs/1809.10756 (2018).
Baydin, A. G. et al. Etalumis: bringing probabilistic programming to scientific simulators at scale. In Proc. International Conference for High Performance Computing, Networking, Storage and Analysis (2019).
Gleisberg, T. et al. Event generation with sherpa 1.1. J. High Energy Phys. 2009, 007 (2009).
Article  Google Scholar 
Blei, D. M. Build, compute, critique, repeat: data analysis with latent variable models. Annu. Rev. Stat. Appl. 1, 203–232 (2014).
Google. Machine learning workflow. https://cloud.google.com/mlengine/docs/tensorflow/ml-solutions-overview. Accessed 13 Dec 2020.
Amershi, S. et al. Software engineering for machine learning: a case study. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) (2019).
Ambrosino, R., Buchanan, B., Cooper, G. & Fine, M. J. The use of misclassification costs to learn rule-based decision support models for cost-effective hospital admission strategies. In Proc. Symposium on Computer Applications in Medical Care 304–308 (1995).
Griffith, G. J. et al. Collider bias undermines our understanding of COVID-19 disease risk and severity. Nat. Commun. 11, 1–12 (2020).
Article  Google Scholar 
Pearl, J. Theoretical impediments to machine learning with seven sparks from the causal revolution. In Proc. 11th ACM International Conference on Web Search and Data Mining (2018).
Nguyen, T.-L. et al. Double-adjustment in propensity score matching analysis: choosing a threshold for considering residual imbalance. BMC Med. Res. Methodol. 17, 1–8 (2017).
Eckles, D. & Bakshy, E. Bias and high-dimensional adjustment in observational studies of peer effects. J. Am. Stat. Assoc. 116, 507–517 (2021).
Xu, Y., Mahajan, D., Manrao, L., Sharma, A. & Kiciman, E. Split-treatment analysis to rank heterogeneous causal effects for prospective interventions. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining. 409–417 (2021).
Richens, J. G., Lee, C. M. & Johri, S. Improving the accuracy of medical diagnosis with causal machine learning. Nat. Commun. 11, (2020).
Paleyes, A., Urma, R.-G. & Lawrence, N. Challenges in deploying machine learning: a survey of case studies. In ACM Computing Surveys (CSUR, 2020).
Chernozhukov, V. et al. Double/debiased machine learning for treatment and structural parameters. Econometrics (2018).
Veitch, V. & Zaveri, A. Sense and sensitivity analysis: simple post-hoc analysis of bias due to unobserved confounding. In NeurIPS. 2020.
Jenniskens, P. et al. CAMS: Cameras for Allsky Meteor Surveillance to establish minor meteor showers. Icarus 216, 40–61 (2011).
Article  ADS  Google Scholar 
Ganju, S. et al. Learnings from frontier development lab and SpaceML—AI accelerators for NASA and ESA. Preprint at ArXiv: abs/2011.04776 (2020).
Zoghbi, S. et al. Searching for long-period comets with deep learning tools. In Deep Learning for Physical Science Workshop, NeurIPS (2017).
Jenniskens, P. et al. A survey of southern hemisphere meteor showers. Planet. Space Sci. 154, 21–29 (2018).
Article  ADS  Google Scholar 
Cohn, D., Ghahramani, Z. & Jordan, M. I. Active learning with statistical models. In NIPS (1994).
Gal, Y., Islam, R. & Ghahramani, Z. Deep bayesian active learning with image data. In International Conference on Machine Learning. 1183–1192 (PMLR, 2017).
Sculley, D. et al. Hidden technical debt in machine learning systems. In NIPS (2015).
Abrahamsson, P., Salo, O., Ronkainen, J. & Warsta, J. Agile Software Development Methods: Review and Analysis (VTT Technical Research Centre of Finland, VTT Publications 478, Otamedia, 2002).
Kuhrmann, M. et al. Hybrid software and system development in practice: waterfall, scrum, and beyond. In Proc. 2017 International Conference on Software and System Process (2017).
Gelman, A. et al. Bayesian workflow. Preprint at ArXiv: abs/2011.01808 (2020).
Chapman, P. et al. CRISP-DM 1.0: Step-by-step data mining guide. SPSS inc 9, 1–73 (2000).
Hohman, F., Wongsuphasawat, K., Kery, M. B. & Patel, K. Understanding and visualizing data iteration in machine learning. In Proc. 2020 CHI Conference on Human Factors in Computing Systems (2020).
Amershi, S., Cakmak, M., Knox, W. B. & Kulesza, T. Power to the people: the role of humans in interactive machine learning. AI Mag. 35, 105–120 (2014).
Google Scholar 
Breck, E. et al. Data Validation for Machine Learning. In Proceedings of Machine Learning and Systems. 334–347 (2019).
Kumar, R., O’Brien, D. R., Albert, K., Viljöen, S. & Snover, J. Failure modes in machine learning systems. Preprint at ArXiv: abs/1911.11034 (2019).
Raji, I. D. et al. Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing. In Proc. 2020 Conference on Fairness, Accountability, and Transparency (2020).
Miksad, R. & Abernethy, A. Harnessing the power of Real-World Evidence (RWE): a checklist to ensure regulatory-grade data quality. Clin. Pharmacol. Ther. 103, 202–205 (2018).
Article  Google Scholar 
Larson, D. B. et al. Regulatory frameworks for development and evaluation of artificial intelligence-based diagnostic imaging algorithms: summary and recommendations. J. Am. College Radiol. (2020).
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K. & Galstyan, A. A survey on bias and fairness in machine learning. ACM Comput. Surv. 54, 1 – 35 (2019).
Google Scholar 
Ntoutsi, E. et al. Bias in data-driven artificial intelligence systems—An introductory survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 10, e1356 (2020).
Jo, E. & Gebru, T. Lessons from archives: strategies for collecting sociocultural data in machine learning. In Proc. 2020 Conference on Fairness, Accountability, and Transparency (2020).
Wiens, J., Price, W. & Sjoding, M. Diagnosing bias in data-driven algorithms for healthcare. Nat. Med. 26, 25–26 (2020).
Article  CAS  Google Scholar 
Challen, R. et al. Artificial intelligence, bias and clinical safety. BMJ Qual. Saf. 28, 231–237 (2019).
Article  Google Scholar 
Cohen, I. & Globerson, S. The impact of debriefing on future performance of projects. Management 4, 177–192 (2015).
Google Scholar 
Obermeyer, Z., Powers, B., Vogeli, C. & Mullainathan, S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 447–453 (2019).
Article  ADS  CAS  Google Scholar 
Mitchell, M. et al. Model cards for model reporting. In Proc. Conference on Fairness, Accountability, and Transparency (2019).
Rivera, S. C., Liu, X., Chan, A., Denniston, A. K. & Calvert, M. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. Nat. Med. 26, 1351–1363 (2020).
Article  Google Scholar 
Gebru, T. et al. Datasheets for datasets. Communications of the ACM. 64, 86–92 (2021).
Szajnfarber, Z. Managing innovation in architecturally hierarchical systems: three switchback mechanisms that impact practice. IEEE Trans. Eng. Manag. 61, 633–645 (2014).
Article  Google Scholar 
Duijm, N. Recommendations on the use and design of risk matrices. Saf. Sci. 76, 21–31 (2015).
Article  Google Scholar 
Zhou, H. & He, Y. Comparative study of OKR and KPI. In 2018 International Conference On E-Commerce And Contemporary Economic Development (Eced 2018). (DEStech Transactions on Economics Business and Management, 2018).
von Neumann, J. Probabilistic logics and the synthesis of reliable organisms from unreliable components. Autom. Stud. 34, 43–98 (1956).
MathSciNet  Google Scholar 
Hutchinson, B. et al. Towards accountability for machine learning datasets: practices from software engineering and infrastructure. In Proc. 2021 ACM Conference on Fairness, Accountability, and Transparency (2021).
Download references
The authors would like to thank Gur Kimchi, Carl Henrik Ek, and Neil Lawrence for valuable discussions about this project.
Pasteur Labs & ISI, Brooklyn, NY, USA
Alexander Lavin
NASA Frontier Development Lab, Mountain View, CA, USA
Alexander Lavin, Siddha Ganju & James Parr
Spotify, London, England
Ciarán M. Gilligan-Lee
University College London, London, UK
Ciarán M. Gilligan-Lee
WhyLabs, Seattle, WA, USA
Alessya Visnjic
Nvidia, Santa Clara, CA, USA
Siddha Ganju
Massachusetts Institute of Technology, Cambridge, MA, USA
Dava Newman
Unity AI, San Francisco, CA, USA
Sujoy Ganguly & Danny Lange
University of Oxford, Oxford, UK
Atílím Güneş Baydin
Microsoft Research, Bangalore, India
Amit Sharma
Konduit, Tokyo, Japan
Adam Gibson
Salesforce Research, San Francisco, CA, USA
Stephan Zheng
Petuum, Pittsburgh, PA, USA
Eric P. Xing
Carnegie Mellon University, Pittsburgh, PA, USA
Eric P. Xing
NASA Jet Propulsion Lab, Pasadena, CA, USA
Chris Mattmann
Alan Turing Institute, London, UK
Yarin Gal
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
You can also search for this author in PubMed Google Scholar
A.L. conceived of the original ideas and framework, with significant contributions towards improving the framework from all co-authors. A.L. initiated the use of MLTRL in practice, including the neuropathology test case discussed here. C.G.-L. contributed insight regarding causal AI, including the section on the counterfactual diagnosis. C.G.-L. also made significant contributions broadly to the paper, notably in the methods descriptions and extensive paper revisions. Si.G. contributed to the spacecraft test case, along with early insights into the framework definitions. A.V. contributed to the definition of later stages involving deployment (as did A.G.), and comparison with traditional software workflows. Both E.X. and Y.G. provided insights regarding AI in academia, and Y.G. additionally contributed to the uncertainty quantification methods. Su.G. and D.L. contributed to the computer vision test case. A.G.B. contributed to the particle physics test case and significant reviews of the write-up. A.S. contributed insights related to causal ML and AI ethics. D.N. provided valuable feedback on the overall framework, and contributed significantly with the details on “switchback mechanisms”. S.Z. contributed to multiple paper revisions, with emphasis on clarity and applicability to broad ML users and teams. J.P. contributed to multiple paper revisions, and to deploying the systems ML methods broadly in practice for Earth and space sciences—same goes for C.M., with additional feedback overall on the methods. All co-authors discussed the content and contributed to editing the manuscript.
Correspondence to Alexander Lavin.
The authors declare no competing interests.
Nature Communications thanks Neil D. Lawrence, Julian Togelius, and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
Reprints and Permissions
Lavin, A., Gilligan-Lee, C.M., Visnjic, A. et al. Technology readiness levels for machine learning systems. Nat Commun 13, 6039 (2022). https://doi.org/10.1038/s41467-022-33128-9
Download citation
Received: 21 December 2020
Accepted: 02 September 2022
Published: 20 October 2022
DOI: https://doi.org/10.1038/s41467-022-33128-9
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative
By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.
Focus
Advertisement
Nature Communications (Nat Commun) ISSN 2041-1723 (online)
© 2022 Springer Nature Limited
Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

source

There were 173 press releases posted in the last 24 hours and 319,222 in the last 365 days.
The global market for 3D Printing is anticipated to increase USD 12.6 billion in 2021 to USD 34.8 billion by 2026, at a CAGR of 22.5% over the forecast period.
Mahesh Patel
VIRTUOSE MARKET RESERACH PRIVATE LIMITED
+1 917-267-7384
email us here
Visit us on social media:
Facebook
LinkedIn
You just read:

EIN Presswire’s priority is source transparency. We do not allow opaque clients, and our editors try to be careful about weeding out false and misleading content. As a user, if you see something we have missed, please do bring it to our attention. Your help is welcome. EIN Presswire, Everyone’s Internet News Presswire™, tries to define some of the boundaries that are reasonable in today’s world. Please see our Editorial Guidelines for more information.

Follow EIN Presswire
© 1995-2022 Newsmatics Inc. dba EIN Presswire
All Right Reserved.

source

Searching for your content…
Phone
877-269-7890 from 8 AM – 10 PM ET
Contact Cision
877-269-7890
from 8 AM – 10 PM ET
News provided by
Sep 22, 2022, 17:00 ET
Share this article
TORONTO, Sept. 22, 2022 /CNW/ – Cybeats Technologies Inc. (“Cybeats” or the “Company”) is pleased to comment on the memorandum (M-22-18) issued by the White House’s Office of Management and Budget on September 14, 2022 under President Biden’s May 2021 Cybersecurity Executive Order.
The memorandum, intended for the heads of executive departments and agencies, focuses on enhancing the security of the software supply chain through secure software development practices.¹
The memo requires all federal agencies to complete a NIST-approved standardized self-attestation form before using any vendor’s or third-party software, including software renewals and major version changes. It also sets new deadlines for federal agencies with regards to their software inventory processes, communication and attestation processes, as well as organizational training needs. The memo further calls on the Cybersecurity and Infrastructure Security Agency (CISA) and the General Services Administration (GSA) to help develop a program plan for a government-wide central repository where software attestations and artifacts can be stored with mechanisms for information protection and sharing among federal agencies.
“By strengthening our software supply chain through secure software development practices, we are building on the Biden-Harris Administration’s efforts to modernize agency cybersecurity practices, including our federal zero trust strategy, improving our detection and response to threats, and our ability to quickly investigate and recover from cyberattacks,^“2 stated the Federal CISO and Deputy National Cyber Director, Chris DeRusha.
“Following the recent rise of cyber-threats and an increased scrutiny of software supply chains, this memorandum comes at a crucial time for federal agencies and critical infrastructure departments” stated Yoav Raiter, CEO of Cybeats. “Cybeats applauds this memorandum and we will continue to put our efforts towards supporting the development of best practices for software supply chain intelligence and security.”
The full memorandum can be read here:
https://www.whitehouse.gov/wp-content/uploads/2022/09/M-22-18.pdf
The National Institute of Standards and Technology have released a Secure Software Development Framework (SSDF) on recommendations for mitigating the risk of software vulnerabilities. The SSDF Framework provides a core set of high-level secure software development practices that can be integrated into each SDLC implementation. The Framework highlights that “following these practices should help software producers reduce the number of vulnerabilities in released software, mitigate the potential impact of the exploitation of undetected or unaddressed vulnerabilities, and address the root causes of vulnerabilities to prevent future recurrences, and to foster communications with suppliers in acquisition processes and other management activities.^“3
Cybeats SBOM Studio, already deployed commercially, helps companies to achieve compliance with the NIST SP 800-218 SSDF Framework as well as with U.S. and North American cybersecurity regulation at large.
SBOM Studio provides organizations with the capability to efficiently manage SBOM (Software Bill of Materials) and software vulnerabilities, and provides proactive mitigation of risks to their software supply chain. Key product features include robust software supply chain intelligence, universal SBOM document management and repository, continuous vulnerability, threat insights, precise risk management, software license infringement and utilization and SBOM exchange with regulatory authorities, customers and vendors.
Cybeats is a leading software supply chain intelligence technology provider, helping organizations manage risk, meet compliance and secure software from procurement, development through operation. Our platform provides customers with deep visibility and universal transparency into their software supply chain, as a result enables them to increase operational efficiencies and revenue. Cybeats.
Software Made Certain. Website: www.cybeats.com
Except for statements of historic fact, this news release contains certain “forward-looking information” within the meaning of applicable securities law. Forward-looking information is frequently characterized by words such as “plan”, “expect”, “project”, “intend”, “believe”, “anticipate”, “estimate” and other similar words, or statements that certain events or conditions “may” or “will” occur. Forward-looking statements are based on the opinions and estimates at the date the statements are made, and are subject to a variety of risks and uncertainties and other factors that could cause actual events or results to differ materially from those anticipated in the forward-looking statements including, but not limited to delays or uncertainties with regulatory approvals, including that of the CSE.
There are uncertainties inherent in forward-looking information, including factors beyond the Company’s control. There are no assurances that the commercialization plans for the technology described in this news release will come into effect on the terms or time frame described herein. The Company undertakes no obligation to update forward-looking information if circumstances or management’s estimates or opinions should change except as required by law. The reader is cautioned not to place undue reliance on forward-looking statements. Under the parent company, Scryb Inc., company filings are available at sedar.com.
______________________________
¹ https://www.whitehouse.gov/wp-content/uploads/2022/09/M-22-18.pdf 
²  https://governmentciomedia.com/white-house-issues-new-memo-secure-supply-chain
³ https://csrc.nist.gov/publications/detail/sp/800-218/final
SOURCE Cybeats Technologies Inc.
For further information: James Van Staveren, Corporate Development, Phone: 647-244-7229, Email: [email protected]
Also from this source
Cybeats Announces Partnership with Veracode, an Industry-Leading Application Security Firm
877-269-7890
from 8 AM – 10 PM ET

source

Maxar stockholders to receive $53.00 per share in cash, a 129% premium to prior closing price
Maxar to remain U.S.-controlled and operated company following close
Advent brings 35+ year investment track record with significant experience in global security and defense
Transaction will support Maxar to accelerate investment in and development of the Company's next-generation satellite technologies and data insights for its customers
Maxar Technologies (NYSE:MAXR) (TSX:MAXR) ("Maxar" or the "Company"), provider of comprehensive space solutions and secure, precise, geospatial intelligence, today announced that it has entered into a definitive merger agreement to be acquired by Advent International ("Advent"), one of the largest and most experienced global private equity investors, in an all-cash transaction that values Maxar at an enterprise value of approximately $6.4 billion. Advent is headquartered in the United States and has a demonstrable track record as a responsible owner of defense and security businesses. Following the close of the transaction, Maxar will remain a U.S.-controlled and operated company.

This press release features multimedia. View the full release here: https://www.businesswire.com/news/home/20221216005078/en/
Under the terms of the definitive merger agreement, Advent has agreed to acquire all outstanding shares of Maxar common stock for $53.00 per share in cash. The purchase price represents a premium of approximately 129% over Maxar's closing stock price of $23.10 on December 15, 2022, the last full trading day prior to this announcement, an approximately 135% premium to the 60-day volume-weighted average price prior to this announcement, and a premium of approximately 34% over Maxar's 52-week high.
Following the closing of the transaction, Maxar will benefit from the significant resources, operational expertise and capacity for investment provided by Advent. As a private company, Maxar will be able to accelerate investments in next-generation satellite technologies and data insights that are vital to the Company's government and commercial customers, as well as pursue select, strategic M&A to further enhance the Company's portfolio of solutions. This includes supporting the successful delivery of the new Legion satellite constellation, accelerating the launch of Legion 7 and 8 satellites and further growing the Earth Intelligence and Space Infrastructure businesses through investments in next-generation capabilities, such as advanced machine learning and 3D mapping. With approximately $28 billion invested across the defense, security and cybersecurity sectors in the last three years, Advent's portfolio companies have substantial expertise supporting many satellite and defense platforms which serve the U.S. government and its allies as well as companies across the globe.
"This transaction delivers immediate and certain value to our stockholders at a substantial premium," said General Howell M. Estes, III (USAF Retired), Chair of Maxar's Board of Directors. "Maxar's mission has never been more important, and this transaction allows us to maximize value for stockholders while accelerating the Company's ability to deliver its mission-critical technology and solutions to customers over the near and long term."
"Today's announcement is an exceptional outcome for stockholders and is a testament to the hard work and dedication of our team, the value Maxar has created and the reputation we have built in our industry," said Daniel Jablonsky, President and CEO of Maxar. "Advent has a proven record of strengthening its portfolio companies and a desire to support Maxar in advancing our long-term strategic objectives. As a private company, we will have enhanced flexibility and additional resources to build on Maxar's strong foundation, further scale operations and capture the significant opportunities in a rapidly expanding market."
"We have tremendous respect and admiration for Maxar, its industry-leading technology and the vital role it serves in supporting the national security of the United States and its allies around the world," said David Mussafer, Chairman and Managing Partner of Advent. "We will prioritize Maxar's commitment as a core provider to the U.S. defense and intelligence communities, and allies, while providing Maxar with the financial and operational support necessary to apply its technology and team members even more fully to the missions and programs of its government and commercial customers."
"In our view, Maxar is a uniquely positioned and attractive asset in satellite manufacturing and space-based high-resolution imagery, with an incredible workforce and many opportunities ahead," said Shonnel Malani, Managing Director and global head of Advent's aerospace and defense team. "We have strong conviction in the growing need for the differentiated solutions Maxar provides, and our goal is to invest in expanding Maxar's satellite constellation as well as supporting Maxar's team to push the boundaries of innovation, ensuring mission success for its customers."
Transaction Details
Under the terms of the agreement, which has been unanimously approved by Maxar's Board of Directors, Maxar stockholders will receive $53.00 in cash for each share of common stock they own.
Advent has arranged committed debt and equity financing commitments for the purpose of financing the transaction, providing a high level of closing certainty. Funds advised by Advent have committed an aggregate equity contribution of $3.1 billion and British Columbia Investment Management Corporation ("BCI") is providing a minority equity investment through a committed aggregate equity contribution equal to $1.0 billion, both on the terms and subject to the conditions set forth in the signed equity commitment letters.
The agreement includes a 60-day "go-shop" period expiring at 11:59 pm EST on February 14, 2023. During this period, the Maxar Board of Directors and its advisors will actively initiate, solicit and consider alternative acquisition proposals from third parties. The Maxar Board will have the right to terminate the merger agreement to enter into a superior proposal subject to the terms and conditions of the merger agreement. There can be no assurance that this "go-shop" will result in a superior proposal, and Maxar does not intend to disclose developments with respect to the solicitation process unless and until it determines such disclosure is appropriate or otherwise required. The Company, Advent and BCI will contemporaneously pursue regulatory reviews and approvals required to conclude the transaction.
The transaction is expected to close mid-2023, subject to customary closing conditions, including approval by Maxar stockholders and receipt of regulatory approvals. The transaction is not subject to any conditionality related to the launch, deployment or performance of Maxar's WorldView Legion satellite program. Upon completion of the transaction, Maxar's common stock will no longer be publicly listed. It is expected that Maxar will continue to operate under the same brand and maintain its current headquarters in Westminster, Colorado.
The foregoing description of the merger agreement and the transactions contemplated thereby is subject to, and is qualified in its entirety by reference to, the full terms of the merger agreement, which Maxar will be filing on Form 8-K.
Advisors
J.P. Morgan Securities LLC is serving as financial advisor to Maxar and Wachtell, Lipton, Rosen & Katz is serving as lead counsel to Maxar. Milbank LLP is serving as Maxar's legal advisor with respect to certain space industry and regulatory matters.
Goldman Sachs & Co. LLC and Morgan Stanley & Co. LLC are serving as financial advisors to Advent and Weil, Gotshal & Manges LLP is serving as lead counsel to Advent. Covington & Burling LLP is serving as Advent's legal advisor with respect to certain regulatory matters.
Skadden, Arps, Slate, Meagher & Flom LLP is serving as lead counsel to BCI. Freshfields Bruckhaus Deringer LLP is serving as BCI's legal advisor with respect to certain regulatory matters.
About Maxar
Maxar Technologies (NYSE:MAXR) (TSX:MAXR) is a provider of comprehensive space solutions and secure, precise, geospatial intelligence. We deliver disruptive value to government and commercial customers to help them monitor, understand and navigate our changing planet; deliver global broadband communications; and explore and advance the use of space. Our unique approach combines decades of deep mission understanding and a proven commercial and defense foundation to deploy solutions and deliver insights with unrivaled speed, scale and cost effectiveness. Maxar's 4,400 team members in over 20 global locations are inspired to harness the potential of space to help our customers create a better world. For more information, visit www.maxar.com .
About Advent International
Founded in 1984 and based in Boston, MA, Advent International is one of the largest and most experienced global private equity investors. The firm has invested in over 400 private equity investments across 41 countries, and as of September 30, 2022, had $89 billion in assets under management. With 15 offices in 12 countries, Advent has established a globally integrated team of over 285 private equity investment professionals across North America, Europe, Latin America and Asia. The firm focuses on investments in five core sectors, including business and financial services; health care; industrial; retail, consumer and leisure; and technology. This includes investments in defense, security and cybersecurity as well as critical national infrastructure.
For over 35 years, Advent has been dedicated to international investing and remains committed to partnering with management teams to deliver sustained revenue and earnings growth for its portfolio companies.
For more information, visit
Website: www.adventinternational.com
LinkedIn: www.linkedin.com/company/advent-international
About BCI
British Columbia Investment Management Corporation (BCI) is amongst the largest institutional investors in Canada with C$211.1 billion under management, as of March 31, 2022. Based in Victoria, British Columbia, with offices in New York City and Vancouver, BCI is invested in: fixed income and private debt; public and private equity; infrastructure and renewable resources; as well as real estate equity and real estate debt. With our global outlook, we seek investment opportunities that convert savings into productive capital that will meet our clients' risk and return requirements over time.
BCI's private equity program actively manages a C$24.8 billion global portfolio of privately-held companies and funds with the potential for long-term growth and value creation. Leveraging our sector-focused teams in business services, consumer, financial services, healthcare, industrials, and technology, media and telecommunications, we work with strategic private equity partners to source and manage direct and co-sponsor/co-investment opportunities.
For more information, please visit bci.ca.
LinkedIn: https://www.linkedin.com/company/british-columbia-investment-management-corporation-bci
Additional Information About the Merger and Where to Find It
This communication relates to the proposed transaction involving Maxar. In connection with the proposed transaction, Maxar will file relevant materials with the U.S. Securities and Exchange Commission (the "SEC"), including Maxar's proxy statement on Schedule 14A (the "Proxy Statement"). This communication is not a substitute for the Proxy Statement or any other document that Maxar may file with the SEC or send to its shareholders in connection with the proposed transaction. BEFORE MAKING ANY VOTING DECISION, SHAREHOLDERS OF MAXAR ARE URGED TO READ ALL RELEVANT DOCUMENTS FILED OR TO BE FILED WITH THE SEC, INCLUDING THE PROXY STATEMENT, WHEN THEY BECOME AVAILABLE BECAUSE THEY WILL CONTAIN IMPORTANT INFORMATION ABOUT THE PROPOSED TRANSACTION. Investors and security holders will be able to obtain the documents (when available) free of charge at the SEC's website, www.sec.gov , or by visiting Maxar's investor relations website, https://investor.maxar.com/overview/default.aspx .
Participants in the Solicitation
Maxar and its directors and executive officers may be deemed to be participants in the solicitation of proxies from the holders of Maxar's common stock in respect of the proposed transaction. Information about the directors and executive officers of Maxar and their ownership of Maxar's common stock is set forth in the definitive proxy statement for Maxar's 2022 Annual Meeting of Stockholders, which was filed with the SEC on March 31, 2022, or its Annual Report on Form 10-K for the year ended December 31, 2021, and in other documents filed by Maxar with the SEC. Other information regarding the participants in the proxy solicitation and a description of their direct and indirect interests, by security holdings or otherwise, will be contained in the Proxy Statement and other relevant materials to be filed with the SEC in respect of the proposed transaction when they become available. Free copies of the Proxy Statement and such other materials may be obtained as described in the preceding paragraph.
Forward-Looking Statements
This communication contains forward-looking statements within the meaning of the Private Securities Litigation Reform Act of 1995, as amended. Statements concerning general economic conditions, our financial condition, including our anticipated revenues, earnings, cash flows or other aspects of our operations or operating results, and our expectations or beliefs concerning future events; and any statements using words such as "believe," "expect," "anticipate," "plan," "intend," "foresee," "should," "would," "could," "may," "estimate," "outlook" or similar expressions, including the negative thereof, are forward-looking statements that involve certain factors, risks and uncertainties that could cause Maxar's actual results to differ materially from those anticipated. Such factors, risks and uncertainties include: (1) the occurrence of any event, change or other circumstances that could give rise to the termination of the merger agreement between the parties to the proposed transaction; (2) the failure to obtain approval of the proposed transaction from Maxar's stockholders; (3) the failure to obtain certain required regulatory approvals or the failure to satisfy any of the other closing conditions to the completion of the proposed transaction within the expected timeframes or at all; (4) risks related to disruption of management's attention from Maxar's ongoing business operations due to the proposed transaction; (5) the effect of the announcement of the proposed transaction on the ability of Maxar to retain and hire key personnel and maintain relationships with its customers, suppliers and others with whom it does business, or on its operating results and business generally; (6) the ability of Maxar to meet expectations regarding the timing and completion of the transaction; (7) the impacts resulting from the conflict in Ukraine or related geopolitical tensions; (8) the impacts of the global COVID-19 pandemic or any other pandemics, epidemics or infectious disease outbreaks; (9) Maxar's ability to generate a sustainable order rate for the satellite and space manufacturing operations and develop new technologies to meet the needs of its customers or potential new customers; (10) the impacts of any changes to the policies, priorities, regulations, mandates and funding levels of governmental entities; (11) the impacts if Maxar's programs fail to meet contractual requirements or its products contain defects or fail to operate in the expected manner; (12) any significant disruption in or unauthorized access to Maxar's computer systems or those of third parties that it utilizes in its operations, including those relating to cybersecurity or arising from cyber-attacks, and security threats could result in a loss or degradation of service, unauthorized disclosure of data, or theft or tampering of intellectual property; (13) satellites are subject to construction and launch delays, launch failures, damage or destruction during launch; (14) if Maxar satellites fail to operate as intended; (15) the impacts of any loss of, or damage to, a satellite and any failure to obtain data or alternate sources of data for Maxar's products; (16) any interruption or failure of Maxar's infrastructure or national infrastructure; (17) Maxar's business with various governmental entities is concentrated in a small number of primary contracts; (18) Maxar operates in highly competitive industries and in various jurisdictions across the world; (19) uncertain global macro-economic and political conditions; (20) Maxar is a party to legal proceedings, investigations and other claims or disputes, which are costly to defend and, if determined adversely to it, could require it to pay fines or damages, undertake remedial measures or prevent it from taking certain actions; (21) Maxar's ability to attract, train and retain employees; (22) any disruptions in U.S. government operations and funding; (23) any changes in U.S. government policy regarding use of commercial data or space infrastructure providers, or material delay or cancellation of certain U.S. government programs; (24) Maxar's business involves significant risks and uncertainties that may not be covered by insurance; (25) Maxar often relies on a single vendor or a limited number of vendors to provide certain key products or services; (26) any disruptions in the supply of key raw materials or components and any difficulties in the supplier qualification process, as well as any increases in prices of raw materials; (27) any changes in Maxar's accounting estimates and assumptions; (28) Maxar may be required to recognize impairment charges; (29) Maxar's business is capital intensive, and it may not be able to raise adequate capital to finance its business strategies, including funding future satellites, or to refinance or renew its debt financing arrangements, or it may be able to do so only on terms that significantly restrict its ability to operate its business; (30) Maxar's ability to obtain additional debt or equity financing or government grants to finance operating working capital requirements and growth initiatives may be limited or difficult to obtain; (31) Maxar's indebtedness and other contractual obligations; (32) Maxar's current financing arrangements contain certain restrictive covenants that impact its future operating and financial flexibility; (33) Maxar's actual operating results may differ significantly from its guidance; (34) Maxar could be adversely impacted by actions of activist stockholders; (35) the price of Maxar's common stock has been volatile and may fluctuate substantially; (36) Maxar's operations in the U.S. government market are subject to significant regulatory risk; (37) failure to comply with the requirements of the National Industrial Security Program Operating Manual could result in interruption, delay or suspension of Maxar's ability to provide its products and services, and could result in loss of current and future business with the U.S. government; (38) Maxar's business is subject to various regulatory risks; (39) any changes in tax law, in Maxar's tax rates or in exposure to additional income tax liabilities or assessments; (40) Maxar's ability to use its U.S. federal and state net operating loss carryforwards and certain other tax attributes may be limited; (41) Maxar's operations are subject to governmental law and regulations relating to environmental matters, which may expose it to significant costs and liabilities; and (42) the other risks listed from time to time in Maxar's filings with the SEC.
For additional information concerning factors that could cause actual results and events to differ materially from those projected herein, please refer to Maxar's Annual Report on Form 10-K for the year ended December 31, 2021 and to other documents filed by Maxar with the SEC, including subsequent Current Reports on Form 8-K and Quarterly Reports on Form 10-Q. Maxar is providing the information in this communication as of this date and assumes no obligation to update or revise the forward-looking statements in this communication because of new information, future events, or otherwise.

View source version on businesswire.com: https://www.businesswire.com/news/home/20221216005078/en/
For Maxar:
Investor Relations
Jonny Bell
(303) 684-5543
jonny.bell@maxar.com
Media Relations
Fernando Vivanco
(720) 877-5220
fernando.vivanco@maxar.com
OR
Scott Bisang / Eric Brielmann / Jack Kelleher
Joele Frank, Wilkinson Brimmer Katcher
(212) 355-4449
dgi-jf@joelefrank.com
For Advent:
Bryan Locke / Jeremy Pelofsky
FGS Global
(212) 687-8080
adventinternational-us@fgsglobal.com
News Provided by Business Wire via QuoteMedia

Maxar Technologies (NYSE:MAXR) (TSX:MAXR), provider of comprehensive space solutions and secure, precise, geospatial intelligence, today announced that the National Oceanic and Atmospheric Administration (NOAA) has modified Maxar's remote sensing license to enable the non-Earth imaging (NEI) capability for its current constellation on orbit as well as its next-generation WorldView Legion satellites.
Through this new license authority, Maxar can collect and distribute images of space objects across the Low Earth Orbit (LEO)—the area ranging from 200 kilometers up to 1,000 kilometers in altitude—to both government and commercial customers. Maxar's constellation is capable of imaging objects at less than 6 inch resolution at these altitudes, and it can also support tracking of objects across a much wider volume of space. Taken together, these capabilities can provide customers with accurate information to assist with mission operations and help address important Space Domain Awareness (SDA) and Space Traffic Management (STM) needs.

"Maxar's NEI capability has been licensed at a pivotal time for the space industry, when the rapid proliferation of space objects is creating an increasingly crowded Low-Earth Orbital environment, creating new risks for government and commercial missions," said Dan Jablonsky, Maxar President and Chief Executive Officer. "Thanks to NOAA's support and hard work, we are now able to leverage our long-held NEI capability to support critical national security missions, help commercial customers better protect and maintain their assets in orbit and provide a new tool to assist with broader space resiliency initiatives."
The ability to provide high-resolution imagery of space objects is more important than ever. There are more than 4,800 active satellites on orbit today, and Euroconsult estimates that 17,000 more satellites will be launched in the next decade. At the same, it is estimated that there are millions of pieces of space debris in LEO, and an impact from even the smallest piece of debris can cause significant damage to a satellite in orbit.
NEI can help address these challenges by bringing more transparency to the near-Earth space domain, thus helping operators better protect and maintain their assets. Maxar will work closely with government and commercial customers to utilize its NEI capabilities to help with a wide range of use cases, including:
The company will begin deploying its NEI capability in 2023 with a select group of early adopters who need to understand and characterize space objects at scale.
To learn more about this capability, visit www.maxar.com/non-earth-imaging .
About Maxar
Maxar Technologies (NYSE:MAXR) (TSX:MAXR) is a provider of comprehensive space solutions and secure, precise, geospatial intelligence. We deliver disruptive value to government and commercial customers to help them monitor, understand and navigate our changing planet; deliver global broadband communications; and explore and advance the use of space. Our unique approach combines decades of deep mission understanding and a proven commercial and defense foundation to deploy solutions and deliver insights with unrivaled speed, scale and cost effectiveness. Maxar's 4,400 team members in over 20 global locations are inspired to harness the potential of space to help our customers create a better world. Maxar trades on the New York Stock Exchange and Toronto Stock Exchange as MAXR. For more information, visit www.maxar.com .
Forward-Looking Statements
This press release may contain forward-looking statements that reflect management's current expectations, assumptions and estimates of future performance and economic conditions. Any such forward-looking statements are made in reliance upon the safe harbor provisions of Section 27A of the Securities Act of 1933 and Section 21E of the Securities Exchange Act of 1934. The Company cautions investors that any forward-looking statements are subject to risks and uncertainties that may cause actual results and future trends to differ materially from those matters expressed in or implied by such forward-looking statements, including those included in the Company's filings with U.S. securities and Canadian regulatory authorities. The Company disclaims any intention or obligation to update or revise any forward-looking statements, whether as a result of new information, future events, or otherwise, other than as may be required under applicable securities law.

View source version on businesswire.com: https://www.businesswire.com/news/home/20221205005197/en/
Investor Relations Contact:
Jonny Bell
Maxar Investor Relations
1-303-684-5543
jonny.bell@maxar.com
Media Contact:
Fernando Vivanco
Maxar Media Relations
1-720-877-5220
fernando.vivanco@maxar.com
News Provided by Business Wire via QuoteMedia

SXM-11 and -12 join SXM-9 and -10 in Maxar development pipeline for SiriusXM
Maxar Technologies (NYSE:MAXR) (TSX:MAXR) and SiriusXM (NASDAQ: SIRI) today announced a new agreement commissioning Maxar to build and deliver two new geostationary communications satellites for SiriusXM.
This press release features multimedia. View the full release here: https://www.businesswire.com/news/home/20221129006004/en/
The Maxar-built SXM-11 and SXM-12 satellites for SiriusXM as shown in an artist rendering. Credit: Maxar.
The SXM-11 and -12 satellite orders increase the total number of spacecraft in development for SiriusXM by Maxar to four, following the 2021 agreement for the construction of SXM-9 and -10.
"This investment reaffirms our commitment to satellite content delivery systems and cutting-edge technology," said Bridget Neville, SiriusXM's Senior Vice President of Satellite and Terrestrial Engineering and Operations. "SXM-11 and -12, along with SXM-9 and -10, will allow us to innovate and improve our service offerings for subscribers and will extend the continuous and reliable delivery of our audio entertainment content."

"This agreement, in combination with SXM-9 and -10 ordered last year, shows one of Maxar's greatest strengths—the advantage of performance at scale," said Chris Johnson, Maxar's Senior Vice President of Space. "These satellites will provide more capability to SiriusXM's fleet, including an expanded service area and higher service quality. We continue to push for new ways to expand capability for commercial geostationary customers, keeping our leadership in this market secure and growing."
There are more than 150 million SiriusXM-equipped vehicles on the road today that rely on SiriusXM's proprietary satellite network, which is also a key delivery mechanism for the company's 360L platform. SiriusXM with 360L combines satellite and streaming to ensure the best possible coverage across the U.S. and Canada and the best customer experience. SiriusXM also offers a suite of satellite-delivered Marine and Aviation services that provide pilots and boaters important weather data and information directly to their cockpits.
SXM-11 and -12 will be twin high-powered digital audio radio satellites, built on Maxar's proven 1300-class platform at the company's manufacturing facilities in Palo Alto and San Jose, California. Maxar has been building satellites for SiriusXM for more than two decades, including the first-generation Sirius satellites launched in 2000; the second-generation Sirius satellites launched in 2009 and 2013; and the company's current third-generation satellites, the first one of which started service in 2021. The delivery of SXM-11 and -12 will bring the number of Maxar-built spacecraft for SiriusXM to 13.
About Maxar
Maxar Technologies is a provider of comprehensive space solutions and secure, precise, geospatial intelligence. We deliver disruptive value to government and commercial customers to help them monitor, understand and navigate our changing planet; deliver global broadband communications; and explore and advance the use of space. Our unique approach combines decades of deep mission understanding and a proven commercial and defense foundation to deploy solutions and deliver insights with unrivaled speed, scale and cost effectiveness. Maxar's 4,400 team members in over 20 global locations are inspired to harness the potential of space to help our customers create a better world. Maxar trades on the New York Stock Exchange and Toronto Stock Exchange as MAXR. For more information, visit www.maxar.com .
About SiriusXM
Sirius XM Holdings Inc. is the leading audio entertainment company in North America, and the premier programmer and platform for subscription and digital advertising-supported audio products. SiriusXM's platforms collectively reach approximately 150 million listeners, the largest digital audio audience across paid and free tiers in North America, and deliver music, talk, news, comedy, entertainment and podcasts. SiriusXM offers the most extensive lineup of professional and college sports in audio. Pandora, a subsidiary of SiriusXM, is the largest ad-supported audio entertainment streaming service in the U.S. SiriusXM's subsidiaries Stitcher, Simplecast and AdsWizz make it a leader in podcast hosting, production, distribution, analytics and monetization. The Company's advertising sales arm, SXM Media, leverages its scale, cross-platform sales organization, and ad tech capabilities to deliver results for audio creators and advertisers. SiriusXM, through Sirius XM Canada Holdings, Inc., also offers satellite radio and audio entertainment in Canada. In addition to its audio entertainment businesses, SiriusXM offers connected vehicle services to automakers. For more about SiriusXM, please go to: www.siriusxm.com .
This communication contains "forward-looking statements" within the meaning of the Private Securities Litigation Reform Act of 1995. Such statements include, but are not limited to, statements about future financial and operating results, our plans, objectives, expectations and intentions with respect to future operations, products and services; and other statements identified by words such as "will likely result," "are expected to," "will continue," "is anticipated," "estimated," "believe," "intend," "plan," "projection," "outlook" or words of similar meaning. Such forward-looking statements are based upon the current beliefs and expectations of our management and are inherently subject to significant business, economic and competitive uncertainties and contingencies, many of which are difficult to predict and generally beyond our control. Actual results and the timing of events may differ materially from the results anticipated in these forward-looking statements.
The following factors, among others, could cause actual results and the timing of events to differ materially from the anticipated results or other expectations expressed in the forward-looking statements: we have been, and may continue to be, adversely affected by supply chain issues as a result of the global semiconductor supply shortage; we face substantial competition and that competition is likely to increase over time; if our efforts to attract and retain subscribers and listeners, or convert listeners into subscribers, are not successful, our business will be adversely affected; we engage in extensive marketing efforts and the continued effectiveness of those efforts is an important part of our business; we rely on third parties for the operation of our business, and the failure of third parties to perform could adversely affect our business; we may not realize the benefits of acquisitions and other strategic investments and initiatives; the ongoing COVID-19 pandemic has introduced significant uncertainty to our business; a substantial number of our Sirius XM service subscribers periodically cancel their subscriptions and we cannot predict how successful we will be at retaining customers; our ability to profitably attract and retain subscribers to our Sirius XM service as our marketing efforts reach more price-sensitive consumers is uncertain; our business depends in part on the auto industry; failure of our satellites would significantly damage our business; our Sirius XM service may experience harmful interference from wireless operations; our Pandora ad-supported business has suffered a substantial and consistent loss of monthly active users, which may adversely affect our Pandora business; our failure to convince advertisers of the benefits of our Pandora ad-supported service could harm our business; if we are unable to maintain revenue growth from our advertising products our results of operations will be adversely affected; changes in mobile operating systems and browsers may hinder our ability to sell advertising and market our services; if we fail to accurately predict and play music, comedy or other content that our Pandora listeners enjoy, we may fail to retain existing and attract new listeners; privacy and data security laws and regulations may hinder our ability to market our services, sell advertising and impose legal liabilities; consumer protection laws and our failure to comply with them could damage our business; failure to comply with FCC requirements could damage our business; if we fail to protect the security of personal information about our customers, we could be subject to costly government enforcement actions and private litigation and our reputation could suffer; interruption or failure of our information technology and communications systems could impair the delivery of our service and harm our business; the market for music rights is changing and is subject to significant uncertainties; our Pandora services depend upon maintaining complex licenses with copyright owners, and these licenses contain onerous terms; the rates we must pay for "mechanical rights" to use musical works on our Pandora service have increased substantially and these new rates may adversely affect our business; failure to protect our intellectual property or actions by third parties to enforce their intellectual property rights could substantially harm our business and operating results; some of our services and technologies may use "open source" software, which may restrict how we use or distribute our services or require that we release the source code subject to those licenses; rapid technological and industry changes and new entrants could adversely impact our services; we have a significant amount of indebtedness, and our debt contains certain covenants that restrict our operations; we are a "controlled company" within the meaning of the NASDAQ listing rules and, as a result, qualify for, and rely on, exemptions from certain corporate governance requirements; while we currently pay a quarterly cash dividend to holders of our common stock, we may change our dividend policy at any time; our principal stockholder has significant influence, including over actions requiring stockholder approval, and its interests may differ from the interests of other holders of our common stock; if we are unable to attract and retain qualified personnel, our business could be harmed; our facilities could be damaged by natural catastrophes or terrorist activities; the unfavorable outcome of pending or future litigation could have an adverse impact on our operations and financial condition; we may be exposed to liabilities that other entertainment service providers would not customarily be subject to; and our business and prospects depend on the strength of our brands. Additional factors that could cause our results to differ materially from those described in the forward-looking statements can be found in our Annual Report on Form 10-K for the year ended December 31, 2021, and our Quarterly Report on Form 10-Q for the quarterly period ended March 31, 2022, which are filed with the Securities and Exchange Commission (the "SEC") and available at the SEC's Internet site ( http://www.sec.gov ). The information set forth herein speaks only as of the date hereof, and we disclaim any intention or obligation to update any forward looking statements as a result of developments occurring after the date of this communication.

View source version on businesswire.com: https://www.businesswire.com/news/home/20221129006004/en/
Kristin Carringer
Maxar Media Relations
1-303-684-4352
kristin.carringer@maxar.com
Kevin Bruns
SiriusXM
Kevin.Bruns@siriusxm.com
News Provided by Business Wire via QuoteMedia

The satellite also known as EchoStar XXIV is expected to launch in the first half of 2023
EchoStar Corporation (Nasdaq: SATS) today announced an amended agreement with Maxar Technologies (NYSE:MAXR) (TSX:MAXR) for production of the EchoStar XXIV satellite, also known as JUPITER™ 3. The satellite, designed for EchoStar's Hughes Network Systems division, is under production at Maxar's facility in Palo Alto, CA. The amended agreement compensates EchoStar for past production delays by providing relief on future payments and expands EchoStar's recourse in the event of any further delays. The satellite is currently planned to launch in the first half of 2023.

"Launching and bringing the Hughes JUPITER 3 satellite into service is our highest priority to meet our customers' needs for connectivity," said Hamid Akhavan , CEO, EchoStar. "This agreement ensures that Maxar shares that priority with us and reinforces our joint commitment to complete production of the satellite to world-class standards, as expeditiously as possible."
"We look forward to continuing our strong collaboration with EchoStar to complete construction of the JUPITER 3 satellite in line with the current schedule," said Daniel Jablonsky , President and CEO, Maxar. "This agreement underscores Maxar's state-of-the-art manufacturing capabilities as we enter into the final phases of construction of this ground-breaking spacecraft."
Once in service, JUPITER 3 will deliver over 500 Gbps of high-throughput satellite capacity, doubling the size of the Hughes JUPITER fleet over North and South America . The satellite will bring ample capacity to grow the company's flagship satellite internet service, HughesNet ^® , and help meet consumer, aeronautical and enterprise demand for more bandwidth and higher speeds.
The satellite is now undergoing final integration in preparation for dynamics testing. Remaining work on the satellite consists of the launch dynamics test, final spacecraft performance tests and shipment to the launch base.
EchoStar Corporation (NASDAQ: SATS) is a premier global provider of satellite communication solutions. Headquartered in Englewood, Colo. , and conducting business around the globe, EchoStar is a pioneer in secure communications technologies through its Hughes Network Systems and EchoStar Satellite Services business segments. For more information, visit www.echostar.com . Follow @EchoStar on Twitter.
Hughes Network Systems, LLC (HUGHES), an innovator in satellite and multi-transport technologies and networks for 50 years, provides broadband equipment and services; managed services featuring smart, software-defined networking; and end-to-end network operation for millions of consumers, businesses, governments and communities worldwide. The Hughes flagship Internet service, HughesNet ^® , connects millions of subscribers across the Americas, and the Hughes JUPITER™ System powers internet access for tens of millions more worldwide. Hughes supplies more than half the global satellite terminal market to leading satellite operators, in-flight service providers, mobile network operators and military customers. A managed network services provider, Hughes supports nearly 500,000 enterprise sites with its HughesON™ portfolio of wired and wireless solutions. Headquartered in Germantown, Maryland , USA, Hughes is owned by EchoStar. To learn more, visit www.hughes.com or follow HughesConnects on Twitter and LinkedIn.
Maxar Technologies (NYSE:MAXR) (TSX:MAXR) is a provider of comprehensive space solutions and secure, precise, geospatial intelligence. We deliver disruptive value to government and commercial customers to help them monitor, understand and navigate our changing planet; deliver global broadband communications; and explore and advance the use of space. Our unique approach combines decades of deep mission understanding and a proven commercial and defense foundation to deploy solutions and deliver insights with unrivaled speed, scale and cost effectiveness. Maxar's 4,400 team members in over 20 global locations are inspired to harness the potential of space to help our customers both create a better world. Maxar trades on the New York Stock Exchange and Toronto Stock Exchange as MAXR. For more information, visit www.maxar.com .
©2022 Hughes Network Systems, LLC, an EchoStar company. Hughes and HughesNet are registered trademarks and JUPITER is a trademark of Hughes Network Systems, LLC.
View original content to download multimedia: https://www.prnewswire.com/news-releases/echostar-and-maxar-amend-agreement-for-hughes-jupiter-3-satellite-production-301685660.html
SOURCE EchoStar Corporation
News Provided by PR Newswire via QuoteMedia

Maxar Technologies (NYSE:MAXR) (TSX:MAXR), provider of comprehensive space solutions and secure, precise, geospatial intelligence, today announced that Galaxy 31 and Galaxy 32, built for Intelsat, are performing as expected after being launched aboard a SpaceX Falcon 9 rocket from Cape Canaveral, Florida.
These two geostationary satellites will enable Intelsat, operator of the world's largest integrated satellite and terrestrial network and leading provider of inflight connectivity, to transfer its services—uninterrupted—as part of the U.S. Federal Communications Commission (FCC) plan to reallocate parts of the C-band spectrum for 5G terrestrial wireless services. Galaxy 31 and Galaxy 32 are the first of five satellites that Intelsat contracted Maxar to build for the C-band transition. All five satellites will be built on Maxar's proven 1300-class platform , which offers the flexibility and power needed for a broad range of customer missions.

Shortly after launch earlier today, both satellites deployed their solar arrays and began receiving and sending signals. Next, Galaxy 31 and Galaxy 32 will begin firing thrusters to commerce their journeys to final geostationary orbit.
"Today's launch of Galaxy 31 and Galaxy 32 is another milestone in Maxar and Intelsat's decades-long relationship," said Chris Johnson, Maxar Senior Vice President and General Manager of Space. "Our team will begin initial on-orbit checkout and Intelsat will proceed with commissioning activities of these satellites so that Intelsat can start moving their services to the new spectrum."
"The Intelsat Galaxy fleet is the most reliable and efficient media content distribution system in North America, enabled by Maxar's engineering and manufacturing expertise," said David C. Wajsgras, Intelsat CEO. "This investment will deliver a high-performance technology path through the next decade."
Maxar also manufactured Intelsat's Galaxy 35 and Galaxy 36, which are preparing for launch in mid-December 2022.
About Maxar
Maxar Technologies (NYSE:MAXR) (TSX:MAXR) is a provider of comprehensive space solutions and secure, precise, geospatial intelligence. We deliver disruptive value to government and commercial customers to help them monitor, understand and navigate our changing planet; deliver global broadband communications; and explore and advance the use of space. Our unique approach combines decades of deep mission understanding and a proven commercial and defense foundation to deploy solutions and deliver insights with unrivaled speed, scale and cost effectiveness. Maxar's 4,400 team members in over 20 global locations are inspired to harness the potential of space to help our customers create a better world. Maxar trades on the New York Stock Exchange and Toronto Stock Exchange as MAXR. For more information, visit www.maxar.com .
Forward-Looking Statements
This press release may contain forward-looking statements that reflect management's current expectations, assumptions and estimates of future performance and economic conditions. Any such forward-looking statements are made in reliance upon the safe harbor provisions of Section 27A of the Securities Act of 1933 and Section 21E of the Securities Exchange Act of 1934. The Company cautions investors that any forward-looking statements are subject to risks and uncertainties that may cause actual results and future trends to differ materially from those matters expressed in or implied by such forward-looking statements, including those included in the Company's filings with U.S. securities and Canadian regulatory authorities. The Company disclaims any intention or obligation to update or revise any forward-looking statements, whether as a result of new information, future events, or otherwise, other than as may be required under applicable securities law.

View source version on businesswire.com: https://www.businesswire.com/news/home/20221112005055/en/
Investor Relations Contact:
Jonny Bell
Maxar Investor Relations
1-303-684-5543
jonny.bell@maxar.com

Media Contact:
Kristin Carringer
Maxar Media Relations
1-303-684-4352
kristin.carringer@maxar.com
News Provided by Business Wire via QuoteMedia

The robotics industry is one of the largest markets in the technology space today, with applications across diverse sectors. However, this diversity may leave market watchers wondering how to invest in robotics.
In simple terms, robotics is defined as the "science and technology behind the design, manufacturing and application of robots." Robots themselves are devices that can perform tasks the same way people do, but without the assistance of human interaction.
Some experts believe a "robot revolution" will completely change the global economy over the next 20 years or so, and with the rise of robotics all but guaranteed, the Investing News Network has put together a primer on the sector. Read on to learn more.

googletag.cmd.push(function() {

var slot = googletag.defineSlot(‘/3404235/inn_com_inarticle’, [[728, 90]]).defineSizeMapping(window.__article_mapping).addService(googletag.pubads());

var div = document.createElement(‘div’);
div.id = slot.getSlotElementId(); // auto-generated by GPT
document.querySelector(‘#gpt_5QM2B7’).appendChild(div);

googletag.enableServices();
googletag.display(slot);
});

According to Market Research Future, the global robotics market is expected to grow at a compound annual growth rate (CAGR) of 22.8 percent between 2021 and 2030 to reach US$214.68 billion. This growth will be tied to the adoption of artificial intelligence (AI) and robotics technology across industries like defense and security, manufacturing, electronics, automotive and healthcare.
Research firm Markets and Markets projects that the industrial segment of the robotics market alone will grow at a CAGR of 14.3 percent from 2022 to 2027 to reach a value of US$30.8 billion. The outlet predicts that the robotics market will play a key role in the coming age of automation, with smart factories increasing demand for robots — in fact, robots are already making their way into consumer goods manufacturing, food processing and packaging and ecommerce supply chain automation.
Demand for industrial robots is also rising in the medical field, including surgical robotics. Grand View Research projects that this segment of the robotics market will experience a CAGR of 19.3 percent from 2022 to 2030 to reach US$18.2 billion.
Aside from that, the automotive industry has long been a sector where industrial robotics has played a hugely transformative role. Not long ago, auto manufacturer BMW (ETR:BMW) signed a supply agreement with robotics firm KUKA (OTC Pink:KUKAF,ETR:KU2) for 5,000 robots to help manufacture BMW's current and future vehicle models.
More recently, in 2021, Nissan (OTC Pink:NSANY,TSE:7201) announced its Intelligent Factory initiative, which will harness AI, the internet of things and robotics technology for vehicle manufacturing to create a zero-emission production system.
For investors looking to enter this emerging tech sector, robotics stocks may be a good place to start.
Stocks are generally the more popular route to take when it comes to investment opportunities, and there's certainly no shortage of robotics stocks to choose from. Major companies in the robotics sector include:
For investors who would rather put their money into the robotics sector as a whole as opposed to a single company, exchange-traded funds (ETFs) may be the way to go. There are a handful of robotics ETFs for investors to choose from, and they track a variety of companies in the industry. Here are three examples to consider:
In summary, the robotics industry isn't going anywhere anytime soon and it looks to have a wealth of investment heading its way. It seems likely to be an attractive space for investors for many years.

This is an updated version of an article originally published by the Investing News Network in 2017.

Don't forget to follow us @INN_Technology for real-time news updates!
Securities Disclosure: I, Melissa Pistilli, hold no direct investment interest in any company mentioned in this article.
Maxar Technologies (NYSE:MAXR) (TSX:MAXR) ("Maxar" or the "Company"), a provider of comprehensive space solutions and secure, precise, geospatial intelligence, today announced financial results for the quarter ended September 30, 2022.
Key points from the quarter include:
"We made good progress in our business during the quarter. In Earth Intelligence, we continue to gain wider traction with the investments we've been making, especially in our 3D and platform capabilities, and are looking forward to the enhanced capacity coming online soon from the WorldView Legion satellites," said Dan Jablonsky, President and Chief Executive Officer. "The Space Infrastructure segment performed well this quarter, generating solid margin expansion and program execution; and continues to be well positioned for wins across national defense, commercial and civil missions."

"We generated positive free cash flow in the quarter and book-to-bill now stands at 1.8x on a year-to-date basis, driven by solid awards at both Earth Intelligence and Space Infrastructure," said Biggs, Porter, Chief Financial Officer. "With Legion nearing launch, our existing backlog and the growth we expect from our diverse and expanding product offerings, we remain committed to substantial growth in earnings and free cash flow next year and over the long term. We are maintaining our prior targets for 2023, having only adjusted them for our recent refinancing activity."
Total revenues remained relatively flat and were $436 million for the three months ended September 30, 2022, compared to $437 million for the same period of 2021.
For the three months ended September 30, 2022, our net loss was $4 million compared to net income of $14 million for the same period of 2021. The decrease in net income was primarily due to an increase in selling, general and administrative costs of $21 million, an increase in other expenses of $14 million, an increase in interest expense of $5 million and an increase in income tax expense of $5 million. This decrease was partially offset by a decrease in product costs of $19 million within our Space Infrastructure segment and a decrease in depreciation and amortization of $10 million for the three months ended September 30, 2022, compared to the same period of 2021.
For the three months ended September 30, 2022, Adjusted EBITDA was $110 million and Adjusted EBITDA margin was 25.2%. This is compared to Adjusted EBITDA of $113 million and Adjusted EBITDA margin of 25.9 % for the same period of 2021. The decrease was primarily driven by lower Adjusted EBITDA from our Earth Intelligence segment and an increase in corporate and other expenses. The decrease was partially offset by an increase in Adjusted EBITDA from our Space Infrastructure segment. The increase in corporate and other expenses was primarily driven by a $5 million foreign exchange loss for the three months ended September 30, 2022, compared to a $1 million foreign exchange loss for the same period of 2021.
We had total order backlog of $2,955 million as of September 30, 2022 compared to $1,893 million as of December 31, 2021. The increase in backlog was primarily driven by an increase in the Earth Intelligence segment partially offset by a decrease in the Space Infrastructure segment. Our unfunded contract options totaled $2,130 million and $650 million as of September 30, 2022 and December 31, 2021, respectively. Unfunded contract options represent estimated amounts of revenue to be earned in the future from negotiated contracts with unexercised contract options and indefinite delivery/indefinite quantity contracts. Unfunded contract options as of September 30, 2022 were primarily comprised of option years in the EOCL Contract (for the periods June 15, 2027 through June 14, 2032) and other U.S. government contracts. Unfunded contract options as of December 31, 2021 were primarily comprised of the option year in the EnhancedView Contract (September 1, 2022 through July 12, 2023) and other U.S. government contracts. On May 25, 2022, we were awarded the EOCL Contract by the NRO, which is a 10-year contract worth up to $3.24 billion, inclusive of a firm 5-year base contract commitment worth $1.5 billion and options worth up to $1.74 billion. The EOCL Contract transitioned the imagery acquisition requirements previously addressed by the EnhancedView Contract and, with this award, replaces the scope of the EnhancedView Contract with respect to such requirements.
Financial Highlights
In addition to results reported in accordance with U.S. GAAP, we use certain non-GAAP financial measures as supplemental indicators of its financial and operating performance. These non-GAAP financial measures include EBITDA, Adjusted EBITDA and Adjusted EBITDA margin . We believe these supplementary financial measures reflect our ongoing business in a manner that allows for meaningful period-to-period comparisons and analysis of trends in its business.

Three Months Ended

Nine Months Ended

September 30,

September 30,

2022

2021

2022

2021
($ millions, except per share amounts)

Revenues
$
436

$
437

$
1,279

$
1,302

Net (loss) income
$
(4
)

$
14

$
(41
)

$
(25
)
EBITDA ¹

94

112

291

311

Total Adjusted EBITDA ¹

110

113

313

312

Net (loss) income per common share:

Basic
$
(0.05
)

$
0.19

$
(0.56
)

$
(0.36
)
Diluted
$
(0.05
)

$
0.19

$
(0.56
)

$
(0.36
)

Weighted average number of common shares outstanding (millions) :

Basic

74.3

72.6

73.8

69.9

Diluted

74.3

74.7

73.8

69.9

¹ This is a non-GAAP financial measure. Refer to section "Non-GAAP Financial Measures" in this earnings release.
Revenues by segment were as follows:

Three Months Ended

Nine Months Ended

September 30,

September 30,

2022

2021

2022

2021
($ millions)

Revenues:

Earth Intelligence
$
275

$
271

$
810

$
804

Space Infrastructure

186

180

549

541

Intersegment eliminations

(25
)

(14
)

(80
)

(43
)
Total revenues
$
436

$
437

$
1,279

$
1,302

We analyze financial performance by segment, which combine related activities within the Company.

Three Months Ended

Nine Months Ended

September 30,

September 30,
($ millions)
2022

2021

2022

2021

Adjusted EBITDA:

Earth Intelligence
$
115

$
124

$
343

$
362

Space Infrastructure

33

14

71

29

Intersegment eliminations

(10
)

(5
)

(28
)

(17
)
Corporate and other expenses

(28
)

(20
)

(73
)

(62
)
Total Adjusted EBITDA ¹
$
110

$
113

$
313

$
312

¹ This is a non-GAAP financial measure. Refer to section "Non-GAAP Financial Measures" in this earnings release.
Earth Intelligence

Three Months Ended

Nine Months Ended

September 30,

September 30,

2022

2021

2022

2021
($ millions)

Revenues
$
275

$
271

$
810

$
804

Adjusted EBITDA
$
115

$
124

$
343

$
362

Adjusted EBITDA margin (as a % of total revenues)

41.8
%

45.8
%

42.3
%

45.0
%
Revenues from the Earth Intelligence segment increased to $275 million from $271 million, or by $4 million, for the three months ended September 30, 2022, compared to the same period in 2021. The increase was primarily driven by a $15 million increase in revenues from the U.S. government, including $11 million from crisis support services, and a $3 million increase in revenues from international defense and intelligence customers. These increases in revenues were partially offset by a $14 million decrease in revenues from commercial programs primarily driven by revenue recognized from a significant commercial contract in the third quarter of 2021.
Adjusted EBITDA decreased to $115 million from $124 million, or by $9 million, for the three months ended September 30, 2022, compared to the same period of 2021. The decrease was primarily driven by increased spending, including on marketing and sales costs of $5 million, IT costs of $4 million, our ERP project of $3 million and other selling, general and administrative costs partially offset by higher revenues.
Space Infrastructure

Three Months Ended

Nine Months Ended

September 30,

September 30,

2022

2021

2022

2021
($ millions)

Revenues
$
186

$
180

$
549

$
541

Adjusted EBITDA
$
33

$
14

$
71

$
29

Adjusted EBITDA margin (as a % of total revenues)

17.7
%

7.8
%

12.9
%

5.4
%
Revenues from the Space Infrastructure segment increased to $186 million from $180 million, or by $6 million, for the three months ended September 30, 2022, compared to the same period of 2021. Revenues for the three months ended increased primarily as a result of a $4 million increase in revenues from U.S. government contracts and a $2 million increase in revenues from recurring commercial programs.
Adjusted EBITDA in the Space Infrastructure segment increased to $33 million from $14 million, or by $19 million, for the three months ended September 30, 2022, compared to the same period of 2021. The increase was primarily due to higher margins driven by reduced risks on certain programs nearing completion for the three months ended September 30, 2022, compared to the same period of 2021.
Corporate and other expenses
Corporate and other expenses include items such as corporate office costs, regulatory costs, executive and director compensation, foreign exchange gains and losses, retention costs and fees for legal and consulting services.
Corporate and other expenses increased to $28 million from $20 million, or by $8 million, for the three months ended September 30, 2022, compared to the same period in 2021. The increase was primarily driven by a $5 million foreign exchange loss for the three months ended September 30, 2022, compared to a $1 million foreign exchange loss for the same period in 2021. The increase was also driven by a $3 million increase in selling, general and administrative costs.
Intersegment eliminations
Intersegment eliminations are related to projects between our segments, including the construction of our WorldView Legion satellites. Intersegment eliminations increased to $10 million from $5 million, or by $5 million, for the three months ended September 30, 2022, compared to the same period in 2021, primarily related to an increase in intersegment satellite construction activity.
MAXAR TECHNOLOGIES INC.
Unaudited Condensed Consolidated Statements of Operations
(In millions, except per share amounts)

Three Months Ended

Nine Months Ended

September 30,

September 30,

2022

2021

2022

2021
Revenues:

Product
$
161

$
166

$
469

$
498

Service

275

271

810

804

Total revenues

436

437

1,279

1,302

Costs and expenses:

Product costs, excluding depreciation and amortization

125

144

380

448

Service costs, excluding depreciation and amortization

95

93

280

286

Selling, general and administrative

110

89

320

261

Depreciation and amortization

64

74

199

221

Gain on sale of assets

(1
)

—

(1
)

—

Operating income

43

37

101

86

Interest expense, net

30

25

129

127

Other expense (income), net

12

(2
)

7

(6
)
Income (loss) before taxes

1

14

(35
)

(35
)
Income tax expense (benefit)

5

—

6

(10
)
Net (loss) income
$
(4
)

$
14

$
(41
)

$
(25
)

Net (loss) income per common share:

Basic
$
(0.05
)

$
0.19

$
(0.56
)

$
(0.36
)
Diluted
$
(0.05
)

$
0.19

$
(0.56
)

$
(0.36
)
MAXAR TECHNOLOGIES INC.
Unaudited Condensed Consolidated Balance Sheets
(In millions, except per share amounts)

September 30,

December 31,

2022

2021
Assets

Current assets:

Cash and cash equivalents

$
28

$
47

Trade and other receivables, net

399

355

Inventory, net

39

39

Advances to suppliers

27

31

Prepaid assets

32

35

Other current assets

64

22

Total current assets

589

529

Non-current assets:

Orbital receivables, net

348

368

Property, plant and equipment, net

1,036

940

Intangible assets, net

712

787

Non-current operating lease assets

136

145

Goodwill

1,627

1,627

Other non-current assets

109

102

Total assets

$
4,557

$
4,498

Liabilities and stockholders' equity

Current liabilities:

Accounts payable

$
91

$
75

Accrued liabilities

73

43

Accrued compensation and benefits

65

111

Contract liabilities

245

289

Current portion of long-term debt

22

24

Current operating lease liabilities

33

42

Other current liabilities

70

38

Total current liabilities

599

622

Non-current liabilities:

Pension and other postretirement benefits

125

134

Operating lease liabilities

136

138

Long-term debt

2,172

2,062

Other non-current liabilities

64

79

Total liabilities

3,096

3,035

Commitments and contingencies

Stockholders' equity:

Common stock ($0.0001 par value, 240 million common shares authorized; 74.3 million and 72.7 million issued and outstanding at September 30, 2022 and December 31, 2021, respectively)

—

—

Additional paid-in capital

2,256

2,235

Accumulated deficit

(763
)

(720
)
Accumulated other comprehensive loss

(32
)

(53
)
Total Maxar stockholders' equity

1,461

1,462

Noncontrolling interest

—

1

Total stockholders' equity

1,461

1,463

Total liabilities and stockholders' equity

$
4,557

$
4,498

MAXAR TECHNOLOGIES INC.
Unaudited Condensed Consolidated Statements of Cash Flows
(In millions)

Nine Months Ended

September 30,

2022

2021
Cash flows provided by (used in):

Operating activities:

Net loss

$
(41
)

$
(25
)
Adjustments to reconcile net loss to net cash provided by (used in) operating activities:

Depreciation and amortization

199

221

Stock-based compensation expense

35

31

Amortization of debt issuance costs and other non-cash interest expense

12

11

Loss from early extinguishment of debt

53

41

Cumulative adjustment to SXM-7 revenue

—

30

Deferred income tax expense

1

2

Other

11

(3
)
Changes in operating assets and liabilities:

Trade and other receivables, net

(31
)

(33
)
Accounts payable and liabilities

5

(57
)
Contract liabilities

(44
)

(20
)
Other

(9
)

(12
)
Cash provided by operating activities – continuing operations

191

186

Cash used in operating activities – discontinued operations

—

(1
)
Cash provided by operating activities

191

185

Investing activities:

Purchase of property, plant and equipment and development or purchase of software

(226
)

(156
)
Acquisition of investment

(2
)

—

Cash used in investing activities – continuing operations

(228
)

(156
)
Financing activities:

Cash paid to extinguish existing Term Loan B

(1,341
)

—

Proceeds from amendment of Term Loan B, net of discount

1,329

—

Repurchase of 9.75% 2023 Notes, including premium

(537
)

(384
)
Proceeds from issuance of 7.75% 2027 Notes

500

—

Net proceeds from Revolving Credit Facility

125

—

Debt issuance costs paid

(27
)

—

Settlement of securitization liability

(10
)

(9
)
Repayments of long-term debt

(12
)

(7
)
Net proceeds from issuance of common stock

—

380

Other

(10
)

(4
)
Cash provided by (used in) financing activities – continuing operations

17

(24
)
(Decrease) increase in cash, cash equivalents, and restricted cash

(20
)

5

Effect of foreign exchange on cash, cash equivalents, and restricted cash

—

—

Cash, cash equivalents, and restricted cash, beginning of year

48

31

Cash, cash equivalents, and restricted cash, end of period

$
28

$
36

Reconciliation of cash flow information:

Cash and cash equivalents

$
28

$
36

Restricted cash included in prepaid and other current assets

—

—

Total cash, cash equivalents, and restricted cash

$
28

$
36

NON-GAAP FINANCIAL MEASURES
In addition to results reported in accordance with U.S. GAAP, we use certain non-GAAP financial measures as supplemental indicators of our financial and operating performance. These non-GAAP financial measures include EBITDA, Adjusted EBITDA and Adjusted EBITDA margin.
We define EBITDA as earnings before interest, taxes, depreciation and amortization, Adjusted EBITDA as EBITDA adjusted for certain items affecting the comparability of our ongoing operating results as specified in the calculation and Adjusted EBITDA margin as Adjusted EBITDA divided by revenue. Certain items affecting the comparability of our ongoing operating results between periods include restructuring, impairments, insurance recoveries, gain (loss) on sale of assets, (gain) loss on orbital receivables allowance, offset obligation fulfillment and transaction and integration related expense. Transaction and integration related expense includes costs associated with de-leveraging activities, acquisitions and dispositions and the integration of acquisitions. Management believes that exclusion of these items assists in providing a more complete understanding of our underlying results and trends, and management uses these measures along with the corresponding U.S. GAAP financial measures to manage our business, evaluate our performance compared to prior periods and the marketplace, and to establish operational goals. Adjusted EBITDA is a measure being used as a key element of our incentive compensation plan. Our Syndicated Credit Facility also uses Adjusted EBITDA in the determination of our debt leverage covenant ratio. The definition of Adjusted EBITDA in the Syndicated Credit Facility includes a more comprehensive set of adjustments that may result in a different calculation therein.
We believe that these non-GAAP measures, when read in conjunction with our U.S. GAAP results, provide useful information to investors by facilitating the comparability of our ongoing operating results over the periods presented, the ability to identify trends in our underlying business, and the comparison of our operating results against analyst financial models and operating results of other public companies.
EBITDA, Adjusted EBITDA and Adjusted EBITDA margin are not recognized terms under U.S. GAAP and may not be defined similarly by other companies. EBITDA and Adjusted EBITDA should not be considered alternatives to net (loss) income as indications of financial performance or as alternate to cash flows from operations as measures of liquidity. EBITDA and Adjusted EBITDA have limitations as an analytical tool and should not be considered in isolation or as a substitute for our results reported under U.S. GAAP. The table below reconciles our net income to EBITDA and Total Adjusted EBITDA and presents Total Adjusted EBITDA margin for the three and nine months ended September 30, 2022 and 2021.

Three Months Ended

Nine Months Ended

September 30,

September 30,

2022

2021

2022

2021

($ millions)

Net (loss) income

$
(4
)

$
14

$
(41
)

$
(25
)

Income tax expense (benefit)

5

—

6

(10
)

Interest expense, net

30

25

129

127

Interest income

(1
)

(1
)

(2
)

(2
)

Depreciation and amortization

64

74

199

221

EBITDA

$
94

$
112

$
291

$
311

Restructuring

5

—

10

—

Transaction and integration related expense

—

1

1

1

Gain on sale of asset

(1
)

—

(1
)

—

Offset obligation fulfillment

12

—

12

—

Total Adjusted EBITDA

$
110

$
113

$
313

$
312

Adjusted EBITDA:

Earth Intelligence

115

124

343

362

Space Infrastructure

33

14

71

29

Intersegment eliminations

(10
)

(5
)

(28
)

(17
)

Corporate and other expenses

(28
)

(20
)

(73
)

(62
)

Total Adjusted EBITDA

$
110

$
113

$
313

$
312

Net (loss) income margin

(0.9
)
%

3.2

%

(3.2
)
%

(1.9
)
%
Total Adjusted EBITDA margin

25.2

%

25.9

%

24.5

%

24.0

%
Cautionary Note Regarding Forward-Looking Statements
This release contains "forward-looking statements" as defined in Section 27A of the U.S. Securities Act of 1933, as amended, and Section 21E of the U.S. Securities Exchange Act of 1934, as amended. Forward-looking statements usually relate to future events and include statements regarding, among other things, our anticipated revenues, cash flows or other aspects of our operations or operating results. Forward-looking statements are often identified by the words "believe," "expect," "anticipate," "plan," "intend," "foresee," "should," "would," "could," "may," "estimate," "outlook" and similar expressions, including the negative thereof.
These forward-looking statements are based on management's current expectations and assumptions based on information currently known to us and our projections of the future, about which we cannot be certain. Forward-looking statements are subject to various risks and uncertainties which could cause actual results to differ materially from the anticipated results or expectations expressed in this press release. As a result, although we believe we have a reasonable basis for each forward-looking statement contained in this press release, undue reliance should not be placed on the forward-looking statements because the Company can give no assurance that they will prove to be accurate. Risks and uncertainties that could cause actual results to differ materially from current expectations include: risks related to the conflict in Ukraine or related geopolitical tensions; our ability to generate a sustainable order rate for our satellite and space manufacturing operations within our Space Infrastructure segment, including our ability to develop new technologies to meet the needs of existing or potential customers; risks related to our business with various governmental entities, which is subject to the policies, priorities, regulations, mandates and funding levels of such governmental entities; our ability to meet our contractual requirements and the risk that our products contain defects or fail to operate in the expected manner; the risk of any significant disruption in or unauthorized access to our computer systems or those of third parties that we utilize in our operations; the ability of our satellites to operate as intended and risks related to launch delays, launch failures or damage or destruction to our satellites during launch; risks related to the interruption or failure of our infrastructure or national infrastructure; the COVID-19 pandemic and its impact on our business operations, financial performance, results of operations and stock price; and the risk factors set forth in Part II, Item 1A, "Risk Factors" in the Company's Quarterly Report on Form 10-Q for the quarter ended June 30, 2022 and filed with the Securities and Exchange Commission (the "SEC") on August 9, 2022, as such risks and uncertainties may be updated or superseded from time to time by subsequent reports we file with the SEC.
The forward-looking statements contained in this press release speak only as of the date hereof are expressly qualified in their entirety by the foregoing risks and uncertainties. Additional risks and uncertainties not currently known to us or that we currently deem to be immaterial may also materially adversely affect our business, prospects, financial condition, results of operations and cash flows. The Company undertakes no obligation to publicly update or revise any of its forward-looking statements after the date they are made, whether as a result of new information, future events or otherwise, except to the extent required by law.
Unless stated otherwise or the context otherwise requires, references to the terms "Company," "Maxar," "we," "us," and "our" to refer collectively to Maxar Technologies Inc. and its consolidated subsidiaries.
Investor/Analyst Conference Call
Maxar President and Chief Executive Officer, Dan Jablonsky, and Executive Vice President and Chief Financial Officer, Biggs Porter, will host an earnings conference call Thursday, November 3, 2022, reviewing the third quarter results, followed by a question and answer session. The call is scheduled to begin promptly at 3:00 p.m. MT (5:00 p.m. ET).
Investors and participants must register for the call in advance by visiting:
https://conferencingportals.com/event/poKRyurD
After registering, participants will receive dial-in information, a passcode, and registrant ID. At the time of the call, participants must dial in using the numbers in the confirmation email and enter their passcode and ID.
The Conference Call will be webcast live and then archived at:
http://investor.maxar.com/events-and-presentations/default.aspx
A replay of the conference call will also be available from Thursday, November 3, 2022 at 6:00 p.m. MT (8:00 p.m. ET) to Thursday, November 17, 2022 at 9:59 p.m. MT (11:59 p.m. ET) at the following numbers:
Toll free North America: 1-800-770-2030
International Dial-In: 1-647-362-9199
Passcode: 81317#
About Maxar
Maxar Technologies (NYSE:MAXR) (TSX:MAXR) is a provider of comprehensive space solutions and secure, precise, geospatial intelligence. We help government and commercial customers monitor, understand and navigate our changing planet; deliver global broadband communications; and explore and advance the use of space. Our approach combines decades of deep mission understanding and a proven commercial and defense foundation to deploy solutions and deliver insights with speed, scale and cost-effectiveness. Maxar's 4,400 team members in over 20 global locations are inspired to harness the potential of space to help our customers create a better world. Maxar's stock trades on the New York Stock Exchange and Toronto Stock Exchange under the symbol "MAXR". For more information, visit www.maxar.com .

View source version on businesswire.com: https://www.businesswire.com/news/home/20221103006220/en/
Jonny Bell | Investor Relations | 1-303-684-5543 | jonny.bell@maxar.com
Fernando Vivanco | Media Relations | 1-720-877-5220 | fernando.vivanco@maxar.com
News Provided by Business Wire via QuoteMedia

Investing News Network websites or approved third-party tools use cookies. Please refer to the  cookie policy for collected data, privacy and GDPR compliance. By continuing to browse the site, you agree to our use of cookies. 

source

Communication is the bedrock of human relationships and cooperation. And this is no different in the workplace. As we delve deeper into the 21st century, with technological advances, global teams, and remote work, it’s never been more important to ensure you have solid communication management in place. A breakdown in communication can affect employee morale, client satisfaction, and your bottom line. 
In this article, I’ll go over the communication management process, and we’ll see how you can implement it into your business!  
Communication management refers to the flow of information within a company and how that company manages it. To do that, you need to focus on the company’s target audience and plan out specific channels of communication. 
In a company environment, the main modes of this communication are generally email and instant messaging.
But, before implementing a communications management process, it’s essential to first understand the lines of communication within your business. For example, who is communicating with whom, when, and why? From here, you can determine whether the right people are sending, receiving and understanding the right information, at the right time. With these answers in mind, you’ll be able to start putting a strategy together. 
We’ll cover this process a bit later, but for now let’s dive into why communication management is important.
Communication management can help you avoid the pitfalls that come from poorly managed communication channels. More specifically, here are 3 recognizable problems that can manifest in your communication methods: 
With poor communication, employees at your company will likely experience increased misunderstandings. For instance, have you ever sent an email outlining a task you’d like an employee to do, only to realize it wasn’t carried out, or was carried out incorrectly when it’s already too late? 
Email was designed for instantaneous communication over the internet. But it doesn’t really work when it comes to assigning and distributing tasks. It’s hard to filter and sort task-based emails from the myriad of other emails your employees are receiving.
Instead, your communications management plan should include a direct way to assign tasks to employees, such as using project management software. Then, your employees will always have a central line of communication for work-related tasks. 
As a result, employees will have a clear understanding of what they need to do and by when. You’ll enhance accountability and empower your employees with clear instructions and workflows. 
When communication and information within your business is disorganized, customer service will suffer. If your employees can’t find documents, information, or resources to sort out a customer issue, the process will take longer than it needs to. This will likely upset your customer and lead to dissatisfaction.
With a clear communication plan in place, employees will be able to easily and efficiently access the information they need to solve a customer issue. With clear outlines of when to use which communication method, your employees should also know to rather call or set up a meeting with a customer when issues arise.  
And this way, you’ll solve problems faster and prevent unnecessary frustration. Clients will be satisfied, knowing you gave their issue the attention and efficiency it deserves. 
When internal communication is poor, workplace productivity will suffer. Employees won’t have easy and efficient access to people, knowledge, or the resources they need to do their jobs. 
You need to make sure your employees can access important information easily — and this includes being able to contact the people they need. And that’s an important part of a proper communication plan. For instance, don’t attach an important document an employee needs to do their job in a random email. It’ll likely get lost in their inbox. Instead, you should invest in tools that help you organize this information. 
As a result, your employees will be armed with all the tools they need to be productive and get their work done. They also won’t be wasting time looking for things in their disorganized inboxes.
You’ve now discovered some of the top issues with poor communication — and how communication management can help you overcome them. So now, let’s take a deep dive into emails versus IMs and see whether one tool stands above the other.  
Email and instant messaging are two of the primary modes of communication within most businesses. But you shouldn’t view them as either or. Instead, you should use them both as part of a successful communication management strategy. 
Essentially, instant messages can be a fantastic tool to complement email communication. But they can never replace email. Even with communication platforms like Slack or Discord, it’s still important to have email as the primary base.
IM should facilitate your secondary communication. To help your employees, your communication management strategy should clearly outline when you should use email or IMs. This way, you’ll use both tools effectively and streamline communication in your business. 
To help you use these tools effectively, I’ll cover some best practices next.
Effective communication systems allow your employees to gain access to useful and high-quality information. Because of this, the communications manager, the project manager, and the policy managers should always work in sync.
The easiest way to implement these systems is to rely on what your employees are already familiar with. Focusing on good email practices and common IM apps will reduce the time people need to adapt. You can look elsewhere if you’ve exhausted all familiar tools and communication issues still occur. Discover the problem and match it with the tools or practices that solve it.
But despite following these best practices — and using emails and IMs in tandem — you may still have problems in your communication. 
So you’ll need a communication management strategy in place to solve these problems. Your strategy will focus on communication as a whole to plug the gaps left by email and IMs. 
Implementing a good communication management strategy isn’t rocket science. But that’s often the reason people neglect it. Most company managers presume communication will come naturally for any collective, which isn’t the case.
You should focus on the following strategies for an excellent internal communication system.
When it comes to communication management it’s always best to have one person in charge of the process. This way, they’ll have complete oversight of your communication system and be able to identify any gaps and issues. Without one person in charge, issues will likely fall through the cracks and damage productivity and efficiency in your business. 
You should create a communication framework as soon as possible. This framework includes information about how your company should communicate with its employees, clients, suppliers and other important entities. On average, more than five people can’t communicate effectively without it. It should be simple and serve the primary purpose, which is clear and concise communication. You don’t need bells or whistles here, just the bare-bones framework.
Nothing frustrates an employee more than unnecessary meetings eating away at their time to complete work. So it’s important you manage your meetings effectively. This means you should know when to communicate something via email or IM, and when it should rather be a meeting. You should also ensure meetings are scheduled well in advance and that participants know if they need to prepare anything.
Constant monitoring is hard to keep up with. But it’s an important part of communication management. You need to monitor how well your communication plan is working. This will help you to determine what works, what doesn’t, and how this relates to certain teams or employees. Armed with this information, you’ll be able to optimize and streamline your communication plan.
While it might seem like an extra step you don’t have time for, you should archive all emails. In essence, old emails are essential for compliance and legal matters. Someone might request five-year-old emails to show track records, task explanations, and other things you’ve long since forgotten. Luckily, you can use email archiving tools to ensure you save an index of all emails and similar information. As a bonus, you’ll save on hardware space and be able to find any piece of information at a moment’s notice.
Thankfully, a variety of tools can help you with communication management. Next, I’ll go through 4 that have shown promise in recent years to narrow down your search.
Communication management tools (CMT) cover several communication-related key performance indicators. Additionally, they allow for better monitoring, active archiving, and network connection management. Let’s have a look at the 4 best tools on the market. Note I’ve mentioned them in no particular order. 
Adobe Workfront primarily focuses on managing workflow and streamlining projects. It’s a good way for communication managers to always be on top of things, especially in creative projects. Adobe Workfront is clean and easy to use, even for novices. Like the rest of their collection, Workfront can run on virtually any platform.
The downside of the platform is that its apps for iOS and Android are a far cry from their desktop option. This might reduce the product’s accessibility for industries such as sales, shipping, and communication, which need a CMT the most.
Wrike is a system that visually represents teams and the people inside them. It offers an excellent view of all relevant projects, files, and tasks. A top benefit of this tool is that it has an automated monitoring capability.
The downside is that communication isn’t the product’s main focus. If you’re using it for marketing and team management, that will be okay — otherwise, you should consider better options with more concise features.
GFI KerioConnect offers a full suite of internal email messaging and external email campaigns. It’s also natively available for virtually all devices, including mobile and Linux, with top email security and compliance standards.
While it has an exemplary user interface, it’s not as colorful as other options. But, if networking and communication are the core of your business, it’s a great tool to invest in.
But this tool’s instant message feature is very basic — it offers fewer choices than other providers, including free ones like Viber and WhatsApp.
With over 20 million users, Slack is probably the most popular on this list. But it’s also a communication platform first and a communication management tool second. Its advantage is that it’s extremely accessible, and quite popular. It also has tons of integrations so it can fit into any of your existing tools.
But Slack doesn’t have emails. In most cases, you’ll have to monitor and archive emails manually through Gmail or another service, which defies the point of having a tool.
To make your choice easier, check out how the best options stack against each other:
These products are pretty different. Hopefully, you could match my description with your company’s needs. Now I’ll wrap it up and take out some key points.
Communication management ensures that everyone in your business communicates as effectively as possible. Effective communication leads to greater business success and increased productivity. Your employees, clients, and your bottom line will benefit from a robust strategy.  
Email and instant messages are two top communication forms that businesses use. But you should know when to use email and when to use instant messages in your business. Regardless, you still need to implement a proper CM strategy to fill in the gaps where these methods fall short.
Learn more about communications management and similar topics with our FAQ and Resources section below!
While customer communication management (CCM) focuses on relations with the customer and retaining their trust, communication management focuses mainly on intra-company communication and improving the workflow. Your job is to ensure infrastructure works as intended and everyone understands their tasks.
Team management involves organizing tasks, determining the best person for each job, and ensuring tasks are completed on time. The communication manager ensures that the flow of information is going well, and while they might use the same team collaboration tools, the approach is different.
Communication tools, as far as software goes, are safe. But, same as email security, you must have some policies in place to ensure employees practice technical safety and personal information security.
As a communications manager, you can work independently or as part of a team. You will also work with IT Service Management (ITSM) and other sectors to ensure the adequate functioning of the communication infrastructure.
Although it is okay to write an email whenever you have the time, you should always expect an answer tomorrow. Using other productivity tools to find someone available for urgent problems is much preferable.
Explore the best practices and models for organizational change that can help your business.
Discover the top five characteristics of effective business communication.
Learn why voice-based communication is getting back to businesses.
Find out what communication compliance is for Office 365 and how you can get it.
Here are the intricacies of M2M communication and why you should consider it.
Read about the main reasons email communication is still in the lead.
You should monitor your network traffic closely to ensure that devices are working properly. This can also help monitor bad traffic, such as cybercriminals trying…
Read More »
Organizational network analysis (ONA) tools help you understand the flow of information and communication in your business. You can use this information for various decisions,…
Read More »
In this article, you learn how to implement a network management solution (NMS). First, you learn more about the NMS features you need in your…
Read More »
Back in the day, looking after a network was as simple as plugging a few leads into a router. Unfortunately, times have changed. Now, you’re…
Read More »
Your email address will not be published. Required fields are marked *
Name*
Email*
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Δdocument.getElementById( “ak_js_1” ).setAttribute( “value”, ( new Date() ).getTime() );
Join Our Newsletters
Learn about the latest security threats, system optimization tricks, and the hottest new technologies in the industry.
TechGenix reaches millions of IT Professionals every month, empowering them with the answers and tools they need to set up, configure, maintain and enhance their networks.
Copyright © 2022 TechGenix

source