Data Engineering Services – Development Process

Further to my introduction, let us look at a typical Data Warehouse/Business Intelligence development process. This basic understanding is needed to decide whether any model for remote-development and/or distributed development could fit in.

Data Operations
Data Operations is an inter-layer between different groups with your company. They store and analyze information from every line of your business – Product, Sales, Inventory, Marketing, Finance etc.

These act as integrated unit with all business groups. They interact directly with stake holders to understand business flows, logic and purpose. They also work with them closely in helping them consume data – in other words ‘helping find value for business using data’.

Data Team
The data team typically has a business analysts who act as a channel between business groups and technical teams (of data). They have a project and program manager who manages projects and deliverable’s. They also would have one or more data architects who is the custodian of the entire data architecture and infrastructure. And they could have data engineers and reporting engineers, who handle develop programs to process data and do reporting respectively.

Stake Holders
On the business side of this, you have stake holder directly interacting with the data team members. They could be Business Owners, Domain Specialists, Functional Business Analysts or facilitators to the Executive Management of the company.

As you can see, the data team and its operation crisscross most if not all of the different functions of your business.

Development Model
A development model of a data team starts with business requirements (compiled as a BRD – Business Requirement Document or PRD – Product Requirement Document) compiled and driven by the business.

These are analyzed by the data team to prepare Functional and Technical Specifications. Project plan is drawn which includes details on the development plan. These delivery plans are typically iterative and has shorter response/release cycles for businesses to get incremental value.

With the basic understanding of different teams and their purpose, let us understand the Data warehouse Development Life Cycle more.

Data Engineering Services – Introduction.

World is flat said Thomas L Friedman.

Companies big and small are finding ways to do business in this globalized world. Innovation is on the move. Outsourcing, Remote-Development, Distributed-Development have become common words in an enterprise. The domain of Data is no stranger to this model either.

So, if you are running Data warehousing, Business Intelligence programs, how does this model fit in a flat world ?

If you are running a services company, what does it take for you to run such programs, successfully for your end customers/clients?

I intend to explore these. Stay tuned.