|
|
|
DATA WAREHOUSE. PROJECT Mgr.
PREAMBLE:
DATABASE
EXPERTISE, CHRONOLOGY (Last two decades) .- At the beginning of that era when microcomputers were introduced, I founded "Scientific Software" (Sept. 1978, Suffolk County, NY). The first applications I developed were computer graphics programs. Two years later, I resigned from British Airways were I had been a Telecommunications Officer for four years, sold my house on Long Island, NY, and moved to Puerto Rico where I dedicated full time to the development of database applications for "micros". In September of 1981 I formally introduced at a farmer's conference my first database application. It was named "XUNA D.M.I.S." (a Dairy Management Information System). The
DBMS engine in its core, as well as its DMIS application layer, had been both
written In BASIC language (the only one that was
available to micros in those days). We
had very stringent memory requirements, due to the limited capacity of these
early PC platforms. And yet, the application sported: ( a ) full color,
( b ) graphic capabilities, ( c ) efficient high-density record storage
despite low-density storage medium, ( d ) scheduled and pre-defined reporting,
(e) pre-defined queries, ( f ) querying tools and universal report generation capabilities, ( g )
statistics, and... a lot more. Emphasis
was put on ( h ) user-friendliness, or it would not have the appeal to the intended
clientele (dairy farmers). Ten
years later I founded another company ("Animal
Science Software International", Two Rivers, WI),
and in September of 1991 I introduced to farmers in Wisconsin a PC version of
the DMIS, this version was named "Decade
D.M.I.S.". I
developed the database engine in C-language.
Windows was not yet what it was to become, and -furthermore- for future
migration to other non-Windows mid-range platforms I chose "Vitamin
C" to build
the GUI, as well as to create the DMIS application on top of the C-engine that I had
to build (in a DOS platform). Again,
for lack of venture capital (and despite great accolades from farmers and trade
publications) I had to close down operations within a couple of years of having
found these entrepreneurships. I had to also deal with a divorce, and the separation from my
children. During the late 70's I
had worked with radars, and data-communications equipment.
In 1992 I went back to my Telecommunications roots and finished a
M.S. degree on that engineering field at the University of Colorado, Boulder.
Then, I tried to earn a Ph.D. degree in Computer Sciences from Colorado
State University, Fort Collins. But
finances and child-support payments caught up with me, and -at the end of 1993-
I had to abandon those studies after having earned only 1/3 of the total
required credits for that program. Since early 1994 -and until May of 1997- I worked for LODGENET ENTERTAINMENT Inc., in Sioux Falls, SD, creating their international Operations. Hence, I got disconnected from software development. I had been in
Florida for about a year, trying to start an International Consulting and
Import/Export business, when in June of 1998 I accepted an offer by INDEPENDENT TECHNOLOGIES Corp.
(ITC).
They needed a person to start their international
operations, but they wanted me to work in software development first.
Sprint ( the telephone company),
had insinuated to ITC they urgently needed
a Data Warehouse application
to consolidate all their payphone's contracts and the issuance of corresponding
commissions. I
had attended the job interview with a laptop, in which I have loaded the
database software I had created for farmers.
It was running under DOS (with Vitamin-C windows), yet it is a powerful engine
and application.
ITC's President (Daryl Ingalsbe) realized that I could be the one who
could make the "Contracts and Commissions" Project a reality .
But, I was "rusty" I had admitted plainly to them. DATA
WAREHOUSING PROJECT HUMAN RESOURCES.- * Originally, I thought I was going to be
a Project Leader, who would only coordinate
the development efforts. Finalize
the specifications with the Client, form the team, and direct the brain storming
sessions. But ITC was quite
small, we had a total of about 100 persons on the company’s payroll, and that
includes people in all four plants. Of these, perhaps as many as 20 are hardware engineers, and
only 3 were software engineers, myself being counted.
We were told, that there was no budget to hire more software people. * One of the software engineers was
pretty strong with the Visual Fox-Pro API, but did not do programming in C++.
He had just finished migrating and enhancing a main ITC software package
(called Wesroc®
was now a quite modern File-Server FoxPro application.
Unfortunately, the code at this time was still riddled with dozens of minor bugs.
Therefore, this engineer would have his hands full for a while, and could not be of
much help. * The other software engineer was pretty
good in Visual C++ . The problem was that his support to the hardware engineers
was of critical importance (in
our test-sets manufacturing facilities).
He could be very helpful answering quick Visual C++ questions, but
we could not count with him for much more than that.
I have to admit that without his help, it would have taken me a lot
longer to get up to speed with Visual C++. * I was the third software guy.
My strength was the development of database engines in C, as well as
interpreting the business rules and doing the code implementation of the
business application. But I had never used Visual C++.
When C++ was being born -in the late 80’s- I was one of those who
studied it, and to prove it I have a pioneering diploma in C++ (from the
University of Colorado, Fall of 1989). But
other than that, I did not have much real practice on that language,
despite its closeness to C. THE PROJECT.- It is called C&C (meaning Contracts and Commissions). C&C can be looked at as the first stage of a full blown DATA WAREHOUSE system. Hundreds of thousands of revenue/charge streams are loaded into C&C® from several DATA MARTS (such as CDR - Call Detail Reports, and other mainframe repositories). The raw data is "dirty", therefore C&C® has a built in ETL module (Extraction, Transformation and Performance) with automatic cleansing tools. Mainly, it (a) extracts data from banks; (b) validates and tags it (a different commission rate could be applied later to each class), and removes inconsistencies; (c) then logs the purified data on a DATA STORE. Users can now apply its built-in processing tools to calculate the commissions multi-dimensionally (that is, on a phone basis, on a contract basis, or on a payee basis). A contract can have any number of phones. A contract's commission may be payable to none, one, or many payees. Furthermore, a payee can be beneficiary of commissions paid on several contracts (what in relational database terminology we refer to as a many-to-many table-relationship). C&C® offers querying tools to analyze results. It also has many predefined queries and reports. C&C® thus becomes an intricate part of the Division operations, as well as a DSS (Decision Support System) to the Management of this Payphone Division. On a monthly basis, commissions' results are automatically loaded onto several logs (repositories) from where they can be analyzed with data mining tools. C&C® -integrated with our Wesroc® product (a Payphone Management and Administration System)- then becomes a full DATA WAREHOUSE for the Payphone Division. Not just the Sales Department can benefit, but also Marketing, Parts Inventory, Repairs and Box-Collection Dispatching, Product Pricing (commission tiers' homogeneity across geographical areas), and Control of Accounts (Payable and Receivables). THE
REQUIREMENTS.- We had a brainstorming session in Omaha on the
specifications, and we came up with the following resolutions to accomplish the project
goals: 1.
Could
not be done with a file-server, it had to be client-server 2.
Oracle
would be the DBMS engine. 3.
We
would not use that vendor's API (the application program interface called Oracle
Enterprise). Instead we will
use C++ to write the client part of this application and its GUI (Visual C++). 4.
It
will not be a 2-tier DBMS application, limited to work only with an Oracle
engine. It would be an n-tier that
could use middleware and ODBC to be able to connect to any other server. 5.
Since
we did not have more software engineers, the “team” would be
made up of just one
person, the reluctant Juan Xuna. 6.
Then
the boss, Daryl Ingalsbe, added the last requirement: we have to have this by
"yesterday". Meaning:
ready for beta test in 3 months, and operational in 6 months. THINGS I HAD TO DO, QUICKLY .- 1) Become a GUI "button-boy" in
days. 2) Buy a lot of technical books, and read
non-stop for the weeks to come. MY MANY HATS, IN THIS “ONE-PERSON TEAM” OXYMORON.- 1)
Project Lead – Who would commit to aggressive
and achievable milestones and dead lines. 2)
SME (Subject Matter Expert) – Who would
need to expand quickly his knowledge of Telco’s CDR (Call Detail
Reporting), to be able to discuss with the client-company the most subtle
details of application’s usage. 3)
Business Analyst – Who would make sure the business
rules are correctly interpreted and implemented.
And that the program has all the needed features, some of which may not
even have been contemplated on the original specs. 4)
Data Modeler Designer - At ITC we do not
have CASE tools (Computer Aided Software Engineering), such as Popkin or
Rational Software.
Therefore the building of the ERD (Entity Relationship Diagrams)
and all the METADATA had to be done the old
fashion way… by brain , of course, with some help from some utilities built-in
on the Microsoft SDK framework (Systems Development Kit). Please,
see at the bottom of this page what metadata is all about, and what entails. 5)
Data Loader – Although I could count on help
from administrative personnel at the Client's premises, for that support to be
effective it would require from me to be in close
proximity with them. Therefore, I relocated to an office at their headquarters
in Overland Park, KS. During the
first stages of development though, the client had not seen the program yet, and
this developer had to be data loader as well. 6)
Application Programmer – No question.
I had to come up to speed with C++, and acquire adequate dexterity and
knowledge on MFC (Microsoft Foundation Classes) and Visual C++ in no
time. 7)
Application Tester - I would have to do the TPC
Benchmarks (Transaction Processing Performance). Design the tests, and create the test data-sets that would
exercise -if possible- all branching of the design flowchart (that would be a
dream on this case).
And do all the time-consuming analysis of test results.
Indeed, very tedious and labor intensive functions, which would normally
require -a least- one full-time engineer to do these tasks alone. 8)
Technical Writer – Regardless of how friendly the
application is, and how well documented the source code is, a trail of documents
have to be created a long the way (Progress Reports). And
-finally- a
User Manual
would have to be written for the Client to understand how the program operates,
on how it is that it meets and complies to its specifications. 9)
Trainer – When all of the above has been
done, I had to be ready to give training seminars on the program, not
just to users on the Client's premises, but also submit to code reviews
(accolades and criticism) from peers at ITC headquarters, so you can learn from their input, and they can learn from
your achievements. This report could
be extended to cover several pages. I will not do that out of respect and
obligation to my Employer and his Client. But,
having covered the most significant topics –from a Résumé perspective- I
will end here. For operational
details, when called to interviews I might take with me a laptop to show C&C®
running (with a set of
test data ). Thanks for
having read this page.
P.S. What
is METADATA? : METADATA is the data about the data! There are 3 types of METADATA.
Some of the framework utilities used these days may help creating such
documentation. However, the developer still has to analyze the project and come up with the set
of tables (scheme), the tables’ fields, and the relational links among tables.
On a project of this magnitude, simple things such as the naming of these
tables and fields could be of major importance. Unless you strictly follow a
robust naming convention, pretty soon you
could be in a messy and nightmarish imbroglio.
As with many other OLAP applications
(Online Analytical Processing), and popular business applications such as ERPs
(Enterprise Resource Planning – PeopleSoft, SAP, Baan), having a good
naming-convention could also help tremendously the users, and your software
colleagues that may have to continue maintaining what you have created. 1)
Business
Metadata: Data Source Systems, Data
Synonyms, Data Access Restrictions, Valid Values, Business Rules 2)
Database
Metadata: Database names, Tables
names, Column names (fields), … 3)
Application
Metadata: Reports, Queries
explanations. |
|
"LEGACY WEBSITE": Will remain unaltered for the foreseeable future.
Phn: (772) 324-1123 |
Fax: (561) 210-1370 Email:
