Popular
Share this on social media:
Credit: spainter_vfx/Shutterstock
IBM Research has recently announced that its Deep Search toolkit has now been released as open source. Deep Search allows scientists and businesses unstructured data. The organisation has now released Deep Search for Scientific Discovery (DS4SD) making the toolkit more versatile and accessible.
Following the launch of the Generative Toolkit for Scientific Discovery (GT4SD) in March, the availability of DS4SD marks the next progression towards building an Open Science Hub for Accelerated Discovery.
To help achieve this goal, IBM choose to publicly release a key component of the Deep Search Experience, its automatic document conversion service. It allows users to upload documents to inspect a document’s conversion quality. DS4SD has a simple drag-and-drop interface, making it very easy for non-experts. IBM also released deepsearch-toolkit, a Python package, where users can programmatically upload and convert documents in bulk. Users can point to a folder and direct the toolkit to upload the documents, convert them, and ultimately analyse the contents of the text, tables, and figures.
The new toolkit interacts and integrates with existing services, and is available to data scientists and engineers through our Python package.
There is a lot of value in unstructured data for scientific research. Consider IBM’s Project Photoresist, for example: IBM used Deep Search in 2020 to find and synthesise a novel photoacid generator molecule for semiconductor manufacturing. These generators pose environmental risks and IBM wanted to discover a better option. Deep Search can ingest data up to 1,000 times faster and screen the data up to 100 times faster than a manual alternative, which allowed us to identify three candidate photoacid generators by the end of 2020. With our end-to-end, AI-powered workflow, IBM scaled and handled the problem with a speed that human scientists simply cannot match, dramatically accelerating the discovery process.
Deep Search uses AI to collect, convert, curate, and ultimately search huge document collections for information that is too specific for standard search tools to handle. It collects data from public, private, structured, and unstructured sources and leverages state-of-the-art AI methods 3456 to convert PDF documents into easily decipherable JSON format with a uniform schema ideal for today’s data scientists. It then applies dedicated natural language processing and computer vision machine-learning algorithms on these documents and ultimately creates searchable knowledge graphs.
The resulting datasets can help businesses make models and identify key trends that inform their decisions. For example, they could match a target acquisition’s financial performance over the past five years, as well as executive turnover during that time. There are exciting applications for Deep Search in healthcare, climate science, and materials research — anywhere large document collections have to be searched — and Deep Search makes it easier to get started.
Deep Search previously required users to provide their data or documents to be searched. IBM has now added more than 364 million public documents, such as patents and research papers. Commercial users of Deep Search can quickly get started searching this data, adding their own data incrementally.
The public release of our automatic document conversion service is only the first step for DS4SD. New capabilities, such as AI models and high quality data-sources, will be made available in the future.
Credit: Gorodenkoff/Shutterstock
Laboratory informatics tools have continued to converge around requirements for data management and movement within an organisation.
Credit: Larichshutterstock/Shutterstock
A round-up of the latest technologies available to scientists and researchers using HPC
Credit: Carlos Castilla/Shutterstock
A round-up of the latest processing and memory technologies
We have just sent you an email so you can verify your account. Please check your spam or junk folder just in case it’s been delivered there.
Need help signing in?
Issues with signing in? Click here
Don’t have an account? Register now
WorkDynamics was launched in 1998.
You should only be asked to sign in once. Not the case? Click here
Register now to read this article and more for free.
March 29-30, 2023
Convene, 117 W 46th St, New York
April 19-20, 2023
Convene, 117 W 46th St, New York
May 23-24, 2023
Convene 117 West 46th Street, New York
May 23-24, 2023
Convene 117 West 46th Street, New York
View all events >
Get limited access to our industry news, analysis and data, plus regular email updates
Get limited access to our industry news, analysis and data, plus regular email updates
A link has been emailed to you – check your inbox.
Email address not recognised.
Don’t have an account? Click here to register
Copyright PEI Media
Not for publication, email or dissemination
December 21, 2022
CRUD is the acronym for CREATE, READ, UPDATE and DELETE. These terms describe the four essential operations for creating and managing persistent data elements, mainly in relational and NoSQL databases.
This post will describe how CRUD operations are used for data processing. We will also show the issues that sysadmins or DevOps engineers may find when monitoring a database.
As mentioned, CRUD operations are used in persistent storage applications, meaning these applications will keep their data even after the system powers down. These are different from operations on data stored in volatile storage, like Random Access Memory or cache files.
CRUD is extensively used in database applications. This includes Relational Database Management Systems (RDBMS) like Oracle, MySQL, and PostgreSQL. It also includes NoSQL databases like MongoDB, Apache Cassandra, and AWS DynamoDB.
Operations similar to CRUD can be performed on persistent data structures like files. For example, you can create a Microsoft Word document, update it, read it, and even delete it from the file explorer. However, files are not record-oriented (or document-oriented in the case of MongoDB or Couchbase). The CRUD terminology is specifically related to record-oriented operations instead of flat file operations.
The CREATE operation adds a new record to a database. In RDBMS, a database table row is referred to as a record, while columns are called attributes or fields. The CREATE operation adds one or more new records with distinct field values in a table.
The same principle applies to NoSQL databases. If the NoSQL database is document-oriented, then a new document (for example, a JSON formatted document with its attributes) is added to the collection, which is the equivalent of an RDBMS table. Similarly, in NoSQL databases like DynamoDB, the CREATE operation adds an item (which is equivalent to a record) to a table.
READ returns records (or documents or items) from a database table (or collection or bucket) based on some search criteria. The READ operation can return all records and some or all fields.
UPDATE is used to modify existing records in the database. For example, this can be the change of address in a customer database or price change in a product database. Similar to READ, UPDATEs can be applied across all records or only a few, based on criteria.
An UPDATE operation can modify and persist changes to a single field or to multiple fields of the record. If multiple fields are to be updated, the database system ensures they are all updated or not at all. Some big data systems don’t implement UPDATE but allow only a timestamped CREATE operation, adding a new version of the row each time.
DELETE operations allow the user to remove records from the database. A hard delete removes the record altogether, while a soft delete flags the record but leaves it in place. For example, this is important in payroll where employment records need to be maintained even after an employee has left the company.
In RDBMS, CRUD operations are performed through Structure Query Language (SQL) commands.
CRUD operations in NoSQL databases will depend on the language of the specific database platform. For example, the Cassandra CQL looks very similar to SQL. In MongoDB, on the other hand, the operations are performed with built-in functions:
Database developers or DBAs often run CRUD statements manually against the database from a client tool. However, in most production use cases, these statements are embedded within the programming language code. When the program runs, the API for the target database takes the CRUD statement and translates it into the native language of the database.
For example, when an ecommerce site visitor initiates the user registration process, a microservice written in Python or Java may read the input values (such as first name, last name, email, address, and so on), and dynamically build an Oracle PL/SQL command. This statement is then sent to the Oracle driver library, which runs it against the database.
Let’s take the ecommerce store example further.
In an online travel agency, a user can CREATE a booking request, READ available flights for the requested route, and make a purchase. This will UPDATE a list of available seats for the flight and CREATE multiple records in the “itinerary” table. If the user terminates the session halfway, then all rows related to this transaction are DELETEd.
Software operations involving CRUD are usually black-box tested. When the testers perform certain operations, they check the backend database rather than analyzing the code to see if the intended changes were made or the correct data returned. Such testing aims to validate each CRUD operation resulting from various possible user interactions in different scenarios.
Efficient database design is the prerequisite for optimal CRUD operations. Without good database design, CRUD operations can adversely affect the performance of a database.
For example, operations like UPDATE or DELETE require exclusive locks on the rows (and their related resources, like data pages or indexes). Locks ensure that when one more row is modified, they are not available to other processes or users for any CRUD operation. This is to ensure the integrity of the data.
You can’t read a record when it’s being deleted or allow two or more users to update a single record simultaneously. Other types of locks, such as shared locks allow simultaneous READs. Locks can be configured at the database or statement level, and different types of locking will dictate which CRUD operations are allowed and how the CRUD operation will behave.
Needless to say, the type of locking and the number of simultaneous locks due to user sessions will affect the performance of a database. For example, a busy ecommerce site with hundreds or thousands of simultaneous users will have many locks operating at the same time. The result could be slow responsiveness as the user waits for locks to be released.
This performance challenge is why database administrators work to ensure CRUD operations can complete as quickly as possible. This requires query tuning based on feedback from monitoring solutions. Such monitoring solutions can show current database locks, metrics, and logs to help the administrator identify possible bottlenecks.
Falcon LogScale Community Edition (previously Humio) offers a free modern log management platform for the cloud. Leverage streaming data ingestion to achieve instant visibility across distributed systems and prevent and resolve incidents.
Falcon LogScale Community Edition, available instantly at no cost, includes the following:
Get Started Free
Arfan Sharif is a product marketing lead for the Observability portfolio at CrowdStrike. He has over 15 years experience driving Log Management, ITOps, Observability, Security and CX solutions for companies such as Splunk, Genesys and Quest Software. Arfan graduated in Computer Science at Bucks and Chilterns University and has a career spanning across Product Marketing and Sales Engineering.
