To assess the extent of soil contamination with a potentially toxic element, it is important to determine the background concentration of that element in the soil. In this study we decided to investigate the relationship between trace element concentrations in soils and the lithology of their parent rocks. We collected 85 topsoils (0–20 cm) samples from the Petorca, La Ligua, Aconcagua, and Casablanca River basins (Valparaiso region, central Chile). For soil sampling, we chose 11 tributary watersheds free of anthropogenic influence. The areas that contributed to the formation of each alluvial soil were delineated and the parent lithologies were acquired from 1:250,000 geological maps. The results show that the total concentrations of arsenic (As), copper (Cu), lead (Pb), and zinc (Zn) in soil vary considerably among the four river basins studied. Soil concentrations of all elements under study are positively correlated with the presence of andesitic volcanic rocks. Furthermore, soils formed from marine sediments and felsic volcanic rocks had the highest concentrations of Pb and As. The carbonate-bearing marine sediments may have adsorbed As and Pb released by weathering of felsic volcanic rocks found upstream of the study area. In contrast, lithologies dominated by continental sedimentary and plutonic rocks showed negative correlations with the concentrations of As, Cu, Pb, and Zn in the soil. Thus, the differences in soil element concentrations in the watersheds under study were due to the different rock types present in the area. We argue that absolute “one-limit-fits-all” background trace element values may be inadequate to assess the extent of soil contamination. Therefore, it is important to determine background concentrations of soil elements in each river basin in the anthropogenically impacted areas. In contrast, erroneous conclusions can be drawn when background concentrations are determined in one region and then applied in another region that is lithologically different.