On July 13th 1863, the New Your draft riot errupted. Horrors were commited arond the city including the lynching an burning of a Black man and the burning of the Colored Orphan Asylum. Colored Orphan Asylum was managed by the Association for the Benefit of Colored Orphans and located on Fifth Avenue between 42nd and 43rd Street. The building was a massive four stories housing between 600 and 800 Black children. The children had access to and played in the gardens and courtyard that surounded the building.
The board of the Colored Orphan Asylum met a week after the riot on July 25, 1863. The meetig notes from Board Meeting describe the attack on the building:
On the 13th July at 4 PM, an infuriated mob … surrounded the premises of the Asylum and 500 of them entered the house … they deliberately set fire to it … simply because it was the home of unoffending colored orphan children.

African American Orphans and Their Caregivers in the Interior Playroom of the Colored Orphan Asylum, New York City (1860)