I led a critical infrastructure project at The Honest Company to completely modernize our payment processing system. We transitioned from an antiquated CSV-based system with Chase bank to a modern API-driven microservice architecture using Braintree, processing over $5M monthly with zero downtime during the transition and throughout its operation.
This wasn't just a technical upgrade—it was a complete business transformation that eliminated manual operations, reduced operational overhead by 90%, and established patterns for future microservice development across the company. The project required careful stakeholder management, team leadership, and mentoring while maintaining zero business disruption.
The existing payment system was an operational nightmare that threatened business growth:
• Manual CSV uploads to remote servers for Chase bank processing
• Batch processing delays - transactions weren't processed in real-time
• Manual error handling - failed transactions required constant manual intervention
• No real-time visibility into payment status or failures
• High operational overhead from manual file transfers and monitoring
• Growing technical debt that made the system increasingly brittle
• Maintain zero downtime during transition (business was actively processing payments)
• Support both synchronous and asynchronous payment processing
• Provide real-time transaction visibility and error handling
• Reduce operational overhead and eliminate manual intervention
• Ensure scalability for growing transaction volume
• Establish patterns for future microservice development
I led a focused team of 2 senior engineers + 1 junior engineer, taking responsibility for:
• Mentoring the junior engineer throughout the project with structured pair programming
• Collaborating with business stakeholders on requirements and timeline management
• Coordinating with infrastructure teams for deployment and monitoring setup
• Making critical technology decisions that prioritized team success over trendy choices
Chose Rails over Go despite team interest in trying a new language:
• Rationale: Team maintainability and familiar patterns more important than theoretical performance gains
• Consideration: Development speed, debugging capabilities, and long-term maintainability
• Result: Faster development and easier team contribution to ongoing maintenance
Challenge: Convincing leadership to invest in complete rebuild rather than incremental patches.
Solution:
• Prepared detailed cost-benefit analysis showing ongoing operational overhead
• Demonstrated risks of patching the increasingly brittle legacy system
• Presented phased approach to minimize business risk
• Emphasized long-term maintainability and scalability benefits
• Rails API-only application for clean separation of concerns
• RESTful architecture with endpoints for different transaction types
• Custom Braintree client gem to abstract API interactions using the Adapter pattern
• Background job processing with Sidekiq for asynchronous operations
• Redis integration for job queuing and caching
• Comprehensive monitoring with Datadog for performance and error tracking
Challenge: Ensuring data consistency between microservice and main Rails monolith.
Solution:
• Implemented event-driven architecture with internal API calls
• Used idempotency tokens to prevent duplicate transactions
• Added comprehensive logging and reconciliation processes
• Built fallback mechanisms for network issues
PostgreSQL schema included a lean transactions table with fields for order_id, status, amount, and braintree_id, plus a payment_logs table for comprehensive audit trails. Strategic indexing on order_id, braintree_id, and status fields ensured fast lookups and efficient filtering of transaction states, supporting high-throughput operations.
• Parallel system operation during transition to ensure zero downtime
• Comprehensive testing in staging environment with Braintree sandbox
• Phased rollout with careful monitoring at each step
• Fallback procedures documented and tested
• Pair programming sessions to demonstrate API integration patterns
• Code review focused on learning with detailed explanations of architectural decisions
• Gradual responsibility increase as junior engineer gained confidence
• Connected work to career growth by explaining industry-standard patterns
Security was paramount given the sensitive nature of payment data. The system adhered to PCI DSS standards by never storing sensitive card information—Braintree handled tokenization at the client side. API keys and credentials were stored securely in environment variables with encryption at rest, and comprehensive auditing logged all access attempts.
• $5M+ monthly transaction processing with zero downtime
• Eliminated manual CSV operations completely
• Real-time payment processing instead of batch delays
• Automatic error detection and handling reduced manual intervention by ~90%
• Comprehensive monitoring provided immediate visibility into system health
• Zero downtime during launch and throughout tenure
• Sub-second response times for synchronous transactions
• Robust background processing for high-volume asynchronous operations
• 99.9%+ uptime with comprehensive error handling and retry logic
• Junior engineer growth - gained expertise in API design, background processing, and system architecture
• Team confidence in maintaining and extending the system
• Reusable patterns established for future microservice development
Lesson: The "boring" technology choice is often the right one for team success.
Application: Choosing Rails over Go prioritized team maintainability over theoretical performance gains, resulting in faster delivery and easier ongoing maintenance.
Lesson: Successful mentoring requires balancing learning opportunities with project delivery pressure.
Application: Structured pair programming and gradual responsibility increases allowed the junior engineer to grow significantly while maintaining project momentum.
Lesson: When replacing systems handling real money, comprehensive testing and phased rollouts are essential.
Application: Running parallel systems during transition and extensive staging testing ensured zero business disruption during the launch.
Lesson: Technical decisions must be presented in business terms to gain leadership support.
Application: Framing the rebuild in terms of operational costs, risk reduction, and scalability rather than technical elegance secured the necessary resources and timeline.
This project became a template for microservice development at The Honest Company, establishing patterns for:
• API design and documentation standards
• Monitoring and observability practices
• Background job processing architecture
• Cross-system integration strategies
The success of this project demonstrated my ability to lead critical infrastructure changes whiledeveloping team members and managing business risk. It established my reputation as someone who could deliver complex technical projects with measurable business impact.
The payment system continued to operate reliably for years after my departure, processing millions of dollars in transactions without the operational overhead that had previously plagued the business. This project showcased the importance of thoughtful architecture, pragmatic technology choices, and effective team leadership in delivering lasting business value.