Over ten percent of men will be diagnosed with prostate cancer (PCa) during their lifetime. Arising from luminal cells of the prostatic acinus, PCa is influenced by multiple cells in its microenvironment. To expand our knowledge and explore means to prevent and treat the disease, it is important to understand what drives the onset and early stages of PCa. In this study, we developed an agent-based model of a prostatic acinus including its microenvironment, to allow for in silico studying of PCa development. The model was based on prior reports and in-house data of tumor cells co-cultured with Cancer Associated Fibroblasts (CAFs) and pro-tumor and/or anti-tumor macrophages. Growth patterns depicted by the model were pathologically validated on H&E slide images of human PCa specimens. We identified that stochasticity of interactions between macrophages and tumor cells at early stages strongly affect tumor development. Additionally, we discovered that more systematic deviations in tumor development result from a combinatorial effect of the probability of acquiring mutations and the tumor-promoting abilities of CAFs and macrophages. In silico modeled tumors were then compared with 494 cancer patients with matching characteristics, showing strong association between predicted tumor load and patients’ clinical outcome. Our findings suggest that the likelihood of tumor formation depends on a combination of stochastic events and systematic characteristics. While stochasticity cannot be controlled, information on systematic effects may aid the development of prevention strategies tailored to the molecular characteristics of an individual patient.