There are a few things to consider when thinking of colour and accessibility, but it generally comes down to one thing - can the user see what you are intending them to see?
If the user can't make out the text because it falls under the required contrast ratios, it is effectively not there for that user.
Making colours accessible can sometimes be seen as an unwanted constraint on creativity, but there are so many colours available that by seeing accessibility as a design philosophy it both reduces the possibilities to a more manageable number and gives a reason for using a particular shade over another.
For example, Slack used to have a palette of 132 colours until they did an accessibility review which enabled them to reduce it down to a much more manageable 18 colours.
Contrast
One of the most talked about is colour contrast. At its most basic this is making sure that a colour is clearly distinguishable from the surrounding colour so the item is clearly perceivable to the user. For example, making sure a font colour has a good contrast against the page background (anyone else remember that phase of web design where everything was small grey text on a white background?).
Contrast is important not only for users with reduced visual acuity, but good contrast helps when screen brightness is reduced (for example with battery saver options) or when viewing a screen in bright sun.
Colour contrast can be measured as a ratio and this is what you will often cited in accessibility reports or on contrast checkers, but generally you want at least 4.5:1 for text content and 3:1 for non-text content (to meet WCAG AA), but bear in mind these are the minimum and you want to be far exceeding these values (the higher the first number the better as the more clear it will be to the user). The WCAG AAA ratio for text contrast is 7:1, so aim for this instead.
Remember these ratios are also important for the various states of the items, so consider colour contrast when looking at focus and hover states too. You don't want a link which appears to disappear when the user tabs to it, and you also want the various states to be obvious when triggered.
Whilst contrast is most commonly linked to text it is also crucial for interface elements. Well defined borders on things like form inputs ensure users can perceive the element itself, important when there is no other indication (like text for a button).
Also pay attention to mixed colour backgrounds, such as when text is overlaid on images. Even if the component is designed with a more subdued image to allow the text to have good contrast, be aware that as the layout shifts based on the viewport, the position of the image in relation to the text may also move. Where the text once overlaid a contrasting part of the image may no longer be the case and the contrast may have suffered.
Colour as information Colour is also something which shouldn't be used in isolation to convey information - for example saying "credits are shown in green and debits in red", or just using a red outline to mark an error on a form input is not helpful. As not everyone perceives colour in the same way (various forms of colour-blindness are very common) it is important that text labels are also used to present the meaning, and it is the meaning which is referred to rather than the colour.
Similarly, just identifying links with a colour can leave users struggling to identify them, so consider keeping the underlines unless it is in a section of the page where it is obvious it is a link (such as a navigation area).
As such, colour should always be a secondary consideration. The simplest way to check something still makes sense is to view it in greyscale which you can do using Chrome's rendering options in devTools, or in MacOS's colour filters (under Accessibility in System Prefs).
Colour-blindness
Speaking of colour-blindness, there is a misconception that red/green colour-blindness just means you can't 'see' the difference between red and green. In fact it causes issues with any colour which uses red or green as a constituent part, for example purple. So the likelihood is many colours will actually look different to a user with this condition.
Also bear in mind that colour-blindness is not binary but has a spectrum of severity in the different types. There are emulators such as in the Chrome developer tools Rendering panel, but these show examples only (I've not found two emulators which agree on how something would be perceived). What they do show however is how different types of colour-blindness can also affect how effective colour contrast is, another reason why you want to make your contrasts as high as possible.
Bear in mind things other than the very common colour-blindness can affect colour perception. Most operating systems now include bedtime routine modes which change the colours emitted after a certain time to reduce blue-light exposure.
Backgrounds and inverted colours
Colour is sometimes used as a background to help define an area. Even where this background has good contrast against the surrounding area, because background colours are not included when using an inverted colour setting like Windows High Contrast this can be lost and important visual grouping lost.
To avoid this, use a transparent border on the element which will be highlighted in the inverted colour scheme to define the area in an alternate but just as effective way.
Detrimental effects of colour choice
As mentioned, some users have problems using sites which have acres of bright white backgrounds, so look at the possibility of implementing a dark mode as an option. Similarly, some users may struggle with a dark scheme so allow them to choose a lighter option. These can be tied to the operating system's user preferences setting, but should always include a toggle to allow the user user to override this for the individual site if desired.
Pure black on white can be difficult for some users with dyslexia, so look at using off-white and dark grey instead.
Finally, colour can also draw attention and distract or simply be overwhelming, so be aware of how this might affect users.