Senior Data Engineer
San Francisco, CA
Location/City : CA - San Francisco
Area Code : 415
Job Type : 1: Full Time
Id : 23430
23430 Senior Data Engineer San Francisco, CA 94105Company:
In just a few short years, our client has grown into America's largest purchase panel, helping brands and retailers grow via next generation consumer insights. Through their portfolio of proprietary mobile apps that incentivize a representative sample of Americans to take pictures of their receipts, they capture nearly 1 out of every 100 shopping trips made across the country. These shoppers also complete in-the-moment surveys to more fully explain the why behind the buy. With more than 300 million brick-and-mortar and ecommerce receipts captured annually, they ve rapidly become a primary source of real-time, omnichannel behavioral insights for industry leaders across the board.Job Description
The ideal candidate is an experienced data engineer and can serve as a thought leader on the team for organizing structured and unstructured data, managing data pipelines, building data models, and scaling data warehouses (that need daily updates and availability across many countries). Our client ?'s platform leverages a data store that performs heavy computation at runtime, as opposed to pre-materializing datasets. In their industry, this has proven to be a huge business advantage, but becomes a challenge as concurrent user count increases. Here ?'s a comprehensive list of challenges that may present themselves as part of the role: What options exist to scale our data warehouse to support growing concurrency and report complexity?
How would we process and organize a new stream of semi-structured data with 10m events/day?
What solutions should be solved with a MPP data warehouse solutions versus a map-reduce system?
Determine SLA strategy with Product & Business partners for new data sets
When should we employ an ETL strategy versus and ELT strategy? Along with a supporting cast of data engineers and application developers, this senior data warehouse lead will help provide answers and implement the solutions that allow our client to continue to build out the world's largest single-source set of purchase data across brands and retailers.Skills & Experience
BS or MS in Computer Science or equivalent work experience.
4+ years of experience in the data warehouse space.
Expert in SQL, including advanced analytical query.
Experience working with a MPP data warehouse (Redshift).
Experience in ETL design and tools (Talend, Pentaho Kettle, Informatica).
Experience with schema design and data modeling.Exceptional candidates will have
Experience configuring and optimizing Vertica clusters.
Experience in working with map-reduce systems (Hadoop, Apache Spark).
Experience in BI Development (Tableau, MicroStrategy).
Experience with NoSQL Systems (Mongo, Cassandra, DynamoDB).
Experience in data science and building models (R, Python/Pandas).
Experience using terraform and/or ansible for infrastructure deployment.
Experience using variety of Amazon Web Services (EC2, ELB, RDS).Contact:Jon Piggins / Redfish TechnologyExecutive Recruiter, IT Sales & MarketingO: 208-788-8260E: jon[at]redfishtech[dot]comL: www.linkedin.com/in/jonpigginsW: www.redfishtech.comRedfish Technology Building Growth-Mode Tech Companies with Hand-Picked Talent.