Multi-stage stochastic integer linear programs (MSILPs) arise in many practical applications, including logistics planning. We introduce a novel aggregation framework to address MSILPs with mixed-integer state variables and continuous local variables. Our framework imposes additional structure to the integer state variables by leveraging the information of the underlying stochastic process, which is modelled as a Markov chain (MC). We present an exact solution method to the aggregated MSILP, which can also be used in an approximation form to obtain dual bounds and implementable feasible solutions. Moreover, we apply two-stage linear decision rule approximations to obtain high-quality decision policies with significantly reduced computational effort. We test the proposed methodologies in a novel MSILP for hurricane disaster relief logistics planning. We illustrate the effectiveness of the proposed approaches, analyze the trade-offs between various MC-based policies, and extract problem-specific insights from the solution behaviours.