Text and Data Mining (TDM) from HKUST Licensed Material: Home

Elsevier allows a certain amount of TDM to subscribed content.

• Supplies over 40 APIs for Elsevier’s products including Scopus, ScienceDirect, SciVal, PlumX, and others.
• Users need to obtain an API key via Elsevier’s Developer Portal.
• HKUST researchers only have access to the content HKUST subscribed to + Open Access.
• An Object Retrieval API is available for mining images.

• There are limitations on how much and what speed you can harvest for TDM:

  "...there are no hard limits on the number of items that may be downloaded via our API. Nevertheless, a reasonable and customary rate limit remains in place to ensure equal access to the API for all users, and we continue to ask users to use our service responsibly.

  We understand the need to be flexible and continue to monitor usage and consult with researchers. However, we do reserve the right to deactivate any API key if we believe usage is abusive or impacting the stability of our systems." -  Text and data mining FAQs

More info available here: https://www.elsevier.com/open-science/research-data/text-and-data-mining

Text and Data Mining

EIU makes its data available for download and analysis with its “EIU Viewpoint add-in for Excel”. EIU provides a user guide to the Excel Add-in.

The terms of use allow you to do text and data mining, but any use in either university assignments or for publishing in academic articles must cite EIU as the data source. You must only use the Excel Add-in or API to access data for use in text and data mining.

Using AI Tools with EIU data

Generative AI and other AI models or products can be developed using EIU content for non-public research and teaching only and when taking specific content protection measures.

• If you are using a third-party platform as a base for an AI model (e.g. OpenAI GPT models), your models must be trained in a secure and “ring-fenced manner”. “Ringfenced” means information that is self-contained to each organization/AI program and isn’t commingled with the rest of the world/internet.
• HKUST’s enterprise version of ChatGPT (via Azure), which is fenced off from training from the underlying OpenAI model, is OK to use.
• Public/non-paying versions of ChatGPT and other generative AI tools may NOT be used by HKUST users with the data from EIU.

EIU also offers additional and bespoke licences for AI use. If you would like to discuss these, please email them at licensing@eiu.com.

Text & Data Mining with Factiva requires a separate license.

The Library can provide contact person for researchers to ask for quote.

Gale Digital Scholar Lab

• Is a platform for text analysis, data mining, and data visualization.
• Users can create and analyze content sets from Using our licensed content in “Gale Primary Sources” collections,

Content includes:  

Gale (Cengage): Data Mining FAQs

• Archives Unbound collections, including:
  • China : Records of the U.S. Department of State, 1930-1949
  • Chinese Maritime Customs Service : The Customs' gazette, 1869-1913.
  • Chinese Maritime Customs Service Publications, 1859-1949
  • Papers of old Shanghai: Business, Banking, and Insurance. [189-1942]
  • Policing the Shanghai International Settlement, 1894-1945
  • The Chinese Civil War and U.S.-China Relations : Records of the U.S. State Department's Office of Chinese Affairs, 1945-1955.
• Nineteenth Century Collections Online, such as:
  • Asia and the West : Diplomacy and Cultural Exchange
• Gale Primary Sources, such as:
  • The Economist Historical Archive, 1843-2020
  • The Independent historical archive, 1986-2016
  • The Sunday Times historical archive, 1822-2006
  • The Times Digital Archive, 1785-2019

A few of them are excluding due to the copyright holders not grant the right, including Financial Times Historical Archive, 1888-2021 and National Geographic Virtual Library.

• China and the modern world in 7 parts:
  1. Diplomacy and political secrets (1869–1950)
  2. Hong Kong, Britain and China, part I: The Rise and Fall of Empires (1841-1951)
  3. Hong Kong, Britain and China, part II (1965-1993)
  4. Imperial China and the West, part I (1815-1881)
  5.  Imperial China and the West, part II (1865 - 1905)
  6. Missionary, Sinology, and Literary Periodicals (1817-1949)
  7. Records of the Maritime Customs Service of China (1854-1949)
• HKUST has signed an Addendum to the licensing contract.
• For researchers' personal use only.
• Delivers data in a hard drive and the library is responsible to arrange users to access the content in the drive.
• Charges a fee that based on the cost of production and delivery.

Constellate

Provides Text and data mining tools and teaching with JSTOR, Portico, and other IThaka collections.

JSTOR Dataset Services
Anyone can request a dataset through either of the two services below.

• Self-service: limit to 25,000 documents; does not cover full text.
• Large/full-text request: by special request and requires an agreement about the use of the data.

The Nexis Uni subscription from HKUST is good for students to use it for research, but not crawling or downloading large volume of data. 

Lexis-Nexis has a section on their website where you can ask about using or purchasing their "Data as a Service" for larger datasets.

They also have a  LexisNexis Bulk Content API mining personal consultation service.

Project Muse supports TDM with prior approval. Check with library staff to obtain a publisher contact for more details.

Here is the relevant section from their standard journal license:

"...subject to prior notification and approval by Project MUSE, [you may] engage in text processing, which is any kind of analysis of natural language text. MUSE will make appropriate arrangements prior to the start of this activity to account for usage data and ensure continued access for the user. his may include but not be limited to a process by which information may be derived from text by identifying patterns and trends within natural language through text categorization, statistical pattern recognition, concept or sentiment extraction, and the association of natural language with indexing terms…" -  https://about.muse.jhu.edu/librarians/license-review/

Researchers can pay extra to text and data mine ProQuest content that HKUST Library already owns or subscribes to via the ProQuest TDM Studio

"Downloading articles from SAGE Journals for the purposes of text and data mining is expressly permitted in our standard licence agreements and our terms of use for no extra fee. You do not need to ask permission to systematically download articles provided that:

• You only use the articles for non-commercial text and data mining.
• You only download articles to which you have legitimate access, for example if they are open access or part of your institution's subscription. If you cannot view an article on SAGE Journals, you will not be able to download it.
• You respect the following limits when downloading SJ content:
  • 1 request every 6 seconds – Monday to Friday between Midnight and Noon in the "America/Los_Angeles" timezone;
  • 1 request every 2 seconds - Monday to Friday between Noon and Midnight in the "America/Los_Angeles" timezone, and all day Saturday and Sunday."

 - https://journals.sagepub.com/page/policies/text-and-data-mining

Text and Data Mining at Springer Nature

• Offers various APIs to facilitate TDM, e.g. Citations API, SN SciGraph APIs, and more.
• Provides a selection of metadata format such as JATS, Dublin Core, ONIX, or MARC records.
• Supports argumentation mining.

HKUST Library's license with Clarivate (owner of Web of Science) allows creating & using custom data sets:

• For internal, non-commercial purposes only
  1. Use the custom dataset for numerical or statistical analyses of data elements derived from the service
  2. Download the custom dataset for use in your own data analytics and proprietary tools
  3. Index the custom dataset for searching by authorized users and display results of such searches performed
  4. Create derivative databases consisting of the results of (1) to (3)

Limitations: You may not distribute, sublicense or publicize any portion of the custom dataset or derivative databases.

See Clarivate's Product/Service terms (p. 24)

Wiley: Text and Data Mining

• Prefers using Crossref Text and Data Mining Service for TDM.
• Users have to accept the click-through license in order to get an API token, but should check with their library to see if a separate agreement is already in place beforehand.

Text and Data Mining Resources -  by  Reese Manceaux of the Atkins Library at UNCC

Text Mining Resources - Princeton University Library