An Introduction to the Introduction

A long-term friend, Alan Tompkins, died quite suddenly in the Spring of 1999. He left behind a concept for a software product. The product had been implemented in prototype form within the Intelligence Community by cobbling together a number of shareware packages. Alan worked in Army Intelligence as an Officer in the Army Reserves for more than 40 years. His involvement with Army Intelligence, and a life-long career as a computer consultant, gave him skills that were unique in the Intelligence Community. He moved to Washington, DC, a few years ago in order to help bring his skills to bear on Intelligence Community problems.

The prototype Alan created allowed personnel in the Intelligence Community to acquire, organize, and view data acquired from the World Wide Web. The basic idea is that you enter Web information from your browser as records in a database, and then generate an internal Web site, in HTML, to view the data in a variety of ways.  Unclassified, publicly available information such as this is often referred to as Open Source Intelligence.

When Al passed away, two of his friends (Eliot Jardines and Jon Bondy) immediately began working on an implementation of Al’s ideas. We started OS-Mosis, Incorporated (OS for Open Source), and this is our first product, called OS-Mosis Pro. A multi-user version is in the works and will be released by the end of 1999.
 

What Problems Does OS-Mosis Solve?

Sometimes we know we’ve seen some information of interest, but we can’t remember where. Often we remember it was at a Web site, and we know that we’ve bookmarked the site. But even if we are organized enough to create folders in our bookmark area, there are many forces at work that make it difficult for us to find the site we seek.

First off, some sites offer facilities, products, or information that might fit into more than one of the categories we’ve already set up as folders. No matter which folder we place the bookmark into, there is some other folders in which we might search, and not find the particular site we are seeking.

Furthermore, many sites have titles like "Main Page", so even if you find the site in the bookmark area, you can’t tell if it’s the one you want, at least not without revisiting the site.

It would be very useful if you could easily create a database of sites you’ve visited. The database should contain the URL (Web address) and the HTML of the particular Web page. It also should contain so-called "meta data", or in other words, data you enter yourself to characterize the site. The meta-data might include a list of key words, an indication of site interest, an indication of site quality, and your comments.

After the database is created, you would be able to search, sort and view the database content in a variety of ways. At the very least for example, you should be able to find all records containing the word "widget".

This is in essence what OS-Mosis does for you. It does so even when your data is basically unstructured. But that's just the beginning.
 
 

• The Research Problem

Suppose you are engaged on a quest, a search for a specific item of information, or even for a range of information. Perhaps you are considering purchasing a new laptop computer. You want to gather information on features and prices in order to make a decision. You visit Web sites for a variety of manufacturers and distributors.

You visit a large number of Web sites, gathering this information into OS-Mosis. Eventually you have 20 or 30 or even 50 records in the database. How can you comprehend all of this data?

OS-Mosis allows you to view the data in four ways. Each method generates a local Web site (for browsing) appropriate to a particular data set or search.  You can generate sites with 1) all of the records and their contents; 2) an outline of the data, with user-selected sort fields; 3) a table organization of the data; 4) a "hierarchy" Web site, where the data is organized as a "tree" of Web pages, using sort fields you specify.

In addition, you can focus on a subset of the records in the database by specifying a Query. This allows you to focus your attention on specifics even when the database has grown quite large.

One OS-Mosis user stated, "OS-Mosis gave me the hope that I didn't have to find the answer in one sitting, since I wasn't just surfing - I was building something I'd be able to continue working on, and profit from later."

The goal of OS-Mosis is to allow you to acquire data rapidly and efficiently, while enabling you to view and/or print your data in flexible and useful ways.
 

A Quick Discussion, for the Impatient

In order to use OS-Mosis, you must 1) define a new Project, 2) define Fields for the Project, 3) acquire data from the Web browser, and 4) Generate a Site to inspect the data.

1. You can define a new Project by clicking on the Menu button, selecting Add/Select Project, typing in a new Project name, and clicking on Add.
2. You can define Project fields by clicking on the Fields tab in the Project Setup form. Each field requires a name, a Type (more about that down below, if you wish), a length (zero means unlimited), and whether the data is required or not. Click on Save and Close when you are done.
3. Acquire data from your Web browser while the OS-Mosis Toolbar lies on top of the browser window. In general, you highlight text and click on field buttons to copy text from the browser to OS-Mosis.
4. Database data can be used to Generate local Web sites. Outlines, Tables, and networks of Web pages can be Generated in seconds. A Query facility allows you to use subsets of the records in the database.

The Details

If you just want to keep track of "all of the sites you’ve ever visited", you can do this with OS-Mosis, by creating a single OS-Mosis Project.

More often, you will have a series of research questions, and you will put the data for each question in a separate Project. One Project might be a search for a new laptop computer. Another might be collecting all of the information you can on your favorite movie, or movie star. Another might be finding out everything you can on how to build a house. One could even conceive of collecting data on every college in the country, resulting in a very large database.

OS-Mosis consists of three primary activities:

1. Project Setup;
2. Data Acquisition, using the OS-Mosis Toolbar; and
3. Site Generation.

Project Setup

OS-Mosis supports multiple "Projects": each Project represents an independent research effort. You define a Project name, and then the database fields that you will collect for the project.

When OS-Mosis is first run, you will see the Toolbar; click on the Menu button and select Setup. Here is the information for the Worksheet. Only the "fields" are required to use OS-Mosis. Thus, you will then probably want to select the Fields tab.

Fields can be of a variety of types:

1. text copied from a Browser;
2. manually entered (keyed) text;
3. data picked from a list of fixed values (like a list of State abbreviations [CA, CT, VA, MA, Esc]);
4. data multipicked from a list (you can select more than one value [such as from a list of key words]);
5. dates; and
6. images.

Data Acquisition

OS-Mosis provides a stay-on-top Toolbar for collecting data. It is, necessarily, as small as possible. Each field in the database is represented as a button on the Toolbar; when you click on the button, data for that field is acquired in whatever manner you have previously specified.

OS-Mosis acquires data from the application you were using just before you click on an OS-Mosis button (the Source Application). When you first run OS-Mosis, it does not know what application you were using: you will have to switch to the Source Application and then back to OS-Mosis. And if you switch between many applications, take care that you do not accidentally acquire data from the "wrong" application.

If you are "copying" data from a Browser, you would first highlight the text which interests you (click and drag), and then click on the Toolbar button: the text is automatically copied from the Browser into the OS-Mosis Toolbar and Project database.

For some Web browsers, OS-Mosis can automatically acquire and store the URL and the HTML of the site, if you wish. If the word URL at the top of the Toolbar is green, then OS-Mosis has identified the browser and the URL; if it is red, then the URL is not currently accessible.

If the site you are visiting has "frames", you should be careful to click on the part of the browser display that you are interested in before saving the database record. Failing to do this can result in OS-Mosis acquiring the wrong HTML, or even just the frame, which may not be useful to you.

Note that OS-Mosis is not limited to Web browsers. You can acquire data from any other Windows application, be it a word processor or a spread sheet.

Required fields have a box on the button (a check box which is not yet filled in). Fields that have data have a check mark on the button. Thus, if you see any buttons with boxes on them, they are required fields that are not yet filled in, and you have to enter data for them before you are able to successfully store the current database record.

As you add records to the database, the numbers on the right side of the Menu button change. The left number (if present, during editing) is the current record number; the right number is the total number of records. If there is only one number, it is the total number of records.

If you place your mouse cursor on top of any of the field buttons, after a moment the field kind (picked, keyed, date, image, etc.) is displayed, if the field is empty. If there is data in the field, what pops up is the contents of the field. This is an easy way to see whether there is data in the field, and what it is.

You can Expand the Toolbar in order to view the content of each of the database fields. This sacrifices screen real estate but allows you to see each of the acquired field values, as well as a summary tab in which all of the fields are displayed. You can also view the Next and Prior record if you wish.

We have tried to make acquisition of data as easy as possible.
 
 

The Bookmark Problem, Revisited.

Now that you understand something about OS-Mosis fields, you can create a simple Project to manage your bookmarks better. Click on the Menu button and select Add/Select Project. Type in a new Project name, Bookmarks. Click on Add.

When the Project Setup form appears, select the Fields tab. Click on the HTML button to add an HTML field, move up one line, click on the Remove button, and Save and Close the new Setup.

You now have a Project with a single HTML field.

As you wander around the Web, if you encounter a page you wish to recall, simply click on Save. The HTML for that site is saved in the database.

When you wish to find a site, click on Menu, then Generate, enter appropriate data in the Query fields, click on the All tab, and then click on Generate. You will see a list in HTML for all of the sites that meet your query criteria.

OS-Mosis provides another way to find a site that is faster but less flexible. Click on Menu and then Quick Search; enter a single keyword or phrase and then <Enter>; a list of all of the matching database records appear on your screen.

If you wish to, you can add extra fields to this Project, such as Topic, or Title, or Key Words, to help you organize the information without using a Query at all.
 
 

Sample Project

We have provided a small sample Project for you to play with. It is called Actors. Click on the Menu button, then on Add/Select Project. Click on the select box at the top, and select Actors and then click on Select. You will find this project to be an easy way to play with the various Site Generation techniques that are described below.
 
 

Site Generation

With the database configured as you wish and the data collected, you can view the data in a variety of formats as local Web sites. To get to the Generation form, click on the Menu button and then select Generation.

The general process entails you specifying some database fields for use during Generation, and clicking on the Generate button to generate the new Web site. If you wish to just view a site you generated previously, you can click on the Browse button.

For each kind of Site Generated, you must select a Title field. This is the database field that you consider to provide the most useful and unique name for each database record. In the case of a database of plants, it might be the plant name; for colleges, the college name. It usually is not the URL, since this is quite cryptic, even if unique.
 
 

Queries: You can Generate Sites using either all of your database records, or a subset of the records. You use Queries to specify database subsets. To do this, you specify field names, comparison operators, and fixed values, and OS-Mosis finds the database records matching your request. Thus, you can find all records containing "widget" in the "HTML" field, if you wish.
 
 

OS-Mosis supports four kinds of Site Generation:

All: This method of site generation is best used when you wish to provide a record of all of the data you acquired in a particular search. You might want to use this when preparing an appendix for a document describing the search results, or if you have a small research project. A single HTML page is created, that can be quite large if the database is large.

Table: You specify one or two field names, and OS-Mosis generates an HTML table, organizing the database information so that it fits in the table.

Suppose you had a database containing information about film Actors. Then suppose you categorized these actors according to their skills, including Comedy, Action, Stunts, and Romance. The resulting Table might look like this:

Similarly, if you had a field identifying how each actor looked, you could generate the following table:

And, if you specified both fields, you would have a two-dimensional table, with database records categorized by both Skills and by Appearance, like this:

Whether you specify one or two fields, the generated Web site contains one home page, with URLs (Web addresses) pointing to each of the detail pages. This method of site generation is useful when there are only one or two fields of interest; when the text of the URLs is fairly short; and when the total number of database records is not large.

Outline: Outline site generation is similar to Table generation, but while Table generation is limited to two dimensions, Outline generation can handle up to four fields. This is what some of the above tables look like when generated as an Outline:

• Skills = Comedy
• Age = 20's
• Will Smith
• Age = 30's
• Tom Cruise
• Michael Fox
• Age = Middle Aged
• Danny DiVito
• John Lithgow
• Age = Silver
• Steve Martin
• Sean Connery
• Paul Newman
• Dick Van Dyke
• Skills = Action
• Age = 20's
• Will Smith
• Age = 30's
• Tom Cruise
• Age = Silver
• Sean Connery
• Paul Newman
• Skills = Stunts
• Age = 20's
• Will Smith
• Age = Silver
• Sean Connery
• Paul Newman
• Skills = Romance
• Age = Teen
• Leonardo De Caprio
• Age = 20's
• Will Smith
• Age = Silver
• Paul Newman

Hierarchy: As the database becomes either large or complex, the sites generated with the above approaches can become difficult to understand. The "Hierarchy" approach is helpful in these cases. This approach generates a "tree" or "hierarchy" of linked pages, allowing you to navigate around, viewing data subsets.

If one generated a Hierarchy site using the Outline data just above the home page of the site looks like this:

If you clicked on "Comedy", you end up at the following page:

And if you clicked on "Sean Connery", you end up at the following page:

Sean Connery
 

DocNum: 4
DateTime: 1999-07-21 15:29:51
URL: [None]
Name: Sean Connery
Appearance: Simply Stunning
Skills: Comedy; Action; Stunts; Romance
Age: Silver

The easiest way for you to fully understand OS-Mosis Site Generation is to play with it. We have provided the Actors Project you can use for this purpose. You can either Browse directly to the Generated sites, or you can modify the site generation parameters and create sites of your own choosing.
 
 
 

Other OS-Mosis Issues

• Vertical Vs Horizontal Toolbar We have found the preferred Toolbar orientation to be vertical, and the preferred position to be on the right side of the screen, but you can select a horizontal orientation if you wish (click on the Menu button), and you can position the Toolbar wherever you wish.

• On Top. We have found that having the Toolbar stay on top is best, but you can turn this off if you wish (click on the Menu button).

• Add/Select Project. Use this to change from one Project to another, create a new Project, delete an old Project, or copy the structure of an old Project to a new one. Click on the Menu button.

• System Info. Sometimes users encounter problems which are difficult for developers to understand. The System Info menu entry (click on the Menu button) will create a file containing information which might help us understand why your system is behaving differently than the systems on which we tested OS-Mosis. And it might provide useful information to you about your own system.

• Editing Database Records. Inevitably, you will want to modify a database record. This can come about because you discover a typo (either one you made, or one in the underlying data you acquired from a Web site), or because you want to change your mind about comments you made a week before.

• Worksheets. Worksheets can help you organize your research efforts before you start work on a project. See the discussion earlier in this document (under Data Acquisition), and look at all of the content of the Setup form. All of the tabs other than the Fields tab are Worksheet content. The Worksheet is created as a separate page when you choose the Hierarchy method of Web site Generation, for archival purposes. Only "fields" are required for using OS-Mosis and you must enable the "Use Worksheet" function to display the Worksheet on your generated Web site.

OS-Mosis™, OS-Mosis Standard™ and OS-Mosis Pro™ are trademarks of OS-Mosis, Incorporated. All other products mentioned, are registered trademarks or trademarks of their respective companies. Copyright © 1999 OS-Mosis, Incorporated. All rights reserved.